Awesome GitHub RepositoriesWeb Data Extractors

Agent-Reach is an AI agent web gateway and search tool that provides language models with the ability to search and read content from the open web, social media, and community forums without using official APIs. It functions as a routing layer that connects large language models to various internet backends while managing content parsing and connection health. The system enables API-free information retrieval by using open-source backends to extract text and metadata from platforms such as Twitter, Reddit, and YouTube. It converts unstructured website content, RSS feeds, and video transcripts

Colly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks. The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into sp

WebAgent is an autonomous web navigation agent and research system designed to browse the internet and synthesize information to answer complex queries. It functions as a reasoning orchestrator that navigates the web iteratively to perform deep research and extract structured data. The project includes a reinforcement learning training pipeline that generates synthetic interaction datasets for model pre-training and fine-tuning. It employs token-level policy gradients to stabilize training in non-stationary environments and uses a dual-mode inference scaling mechanism to balance execution bet

Este proyecto es una librería de web scraping en Python y una suite de recolección de datos automatizada. Proporciona herramientas para extraer datos estructurados de sitios web, implementar crawlers para navegar enlaces y analizar estructuras DOM HTML para aislar elementos y atributos específicos. El toolkit incluye un pipeline para procesar texto no estructurado y limpiar contenido web crudo para extraer información significativa. También cuenta con capacidades para la extracción de datos de imágenes y la integración de APIs externas para recuperar datos estructurados desde endpoints remotos. El sistema cubre áreas clave como la extracción automatizada de datos web, flujos de trabajo de crawling y técnicas para evitar obstáculos de scraping mediante proxies y resolutores de CAPTCHA.