Awesome GitHub RepositoriesWeb Data Extractors

Agent-Reach is an AI agent web gateway and search tool that provides language models with the ability to search and read content from the open web, social media, and community forums without using official APIs. It functions as a routing layer that connects large language models to various internet backends while managing content parsing and connection health. The system enables API-free information retrieval by using open-source backends to extract text and metadata from platforms such as Twitter, Reddit, and YouTube. It converts unstructured website content, RSS feeds, and video transcripts

Colly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks. The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into sp

WebAgent is an autonomous web navigation agent and research system designed to browse the internet and synthesize information to answer complex queries. It functions as a reasoning orchestrator that navigates the web iteratively to perform deep research and extract structured data. The project includes a reinforcement learning training pipeline that generates synthetic interaction datasets for model pre-training and fine-tuning. It employs token-level policy gradients to stabilize training in non-stationary environments and uses a dual-mode inference scaling mechanism to balance execution bet

Acest proiect este o bibliotecă Python pentru web scraping și o suită de colectare automată a datelor. Oferă instrumente pentru extragerea datelor structurate de pe site-uri web, implementarea de web crawlere pentru a naviga prin link-urile site-urilor și parsarea structurilor DOM HTML pentru a izola elemente și atribute specifice. Toolkit-ul include un pipeline pentru procesarea textului nestructurat și curățarea conținutului web brut pentru a extrage informații relevante. De asemenea, dispune de capabilități pentru extragerea datelor din imagini și integrarea API-urilor externe pentru a prelua date structurate de la endpoint-uri remote. Sistemul acoperă domenii largi de capabilități, inclusiv extragerea automată a datelor web, fluxuri de lucru de web crawling și tehnici pentru a ocoli obstacolele de scraping folosind proxy-uri și rezolvatoare de CAPTCHA.