Awesome GitHub RepositoriesDistributed Crawl Coordination

Firecrawl is a web data extraction platform designed to convert unstructured web content into clean, LLM-ready formats like markdown or JSON. It functions as an autonomous web crawler and scraper, capable of mapping entire domains, performing recursive navigation, and executing complex data gathering tasks. By leveraging headless browser orchestration, the system handles dynamic, JavaScript-heavy pages to ensure comprehensive data capture. The platform distinguishes itself through its focus on agentic workflows, providing a programmatic interface that allows autonomous agents to perform live

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Pholcus es un framework de web crawler distribuido escrito en Go, diseñado para la extracción de datos de alta concurrencia. Funciona como un orquestador de rastreo distribuido y un motor de extracción de datos dinámicos, utilizando una arquitectura servidor-cliente para coordinar tareas a través de múltiples nodos. El sistema integra un motor de navegador headless para renderizar contenido dinámico y ejecutar JavaScript, permitiéndole extraer datos de aplicaciones de una sola página (SPA). Cuenta con una interfaz de gestión basada en web para configurar parámetros de arañas y monitorear el progreso de la ejecución, junto con la capacidad de actualizar reglas de extracción mediante archivos de configuración con recarga en caliente (hot-reloading) sin reiniciar el sistema. La gestión del tráfico se maneja mediante la rotación de pools de proxies y la aleatorización de solicitudes para evadir la detección de bots y evitar límites de tasa. El framework incluye recuperación de puntos de control basada en estado para reanudar tareas tras fallos y proporciona adaptadores de almacenamiento conectables para exportar datos extraídos a bases de datos, colas de mensajes o archivos.

Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica