Awesome GitHub RepositoriesDistributed Crawl Coordination

Firecrawl is a web data extraction platform designed to convert unstructured web content into clean, LLM-ready formats like markdown or JSON. It functions as an autonomous web crawler and scraper, capable of mapping entire domains, performing recursive navigation, and executing complex data gathering tasks. By leveraging headless browser orchestration, the system handles dynamic, JavaScript-heavy pages to ensure comprehensive data capture. The platform distinguishes itself through its focus on agentic workflows, providing a programmatic interface that allows autonomous agents to perform live

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Pholcus este un framework de web crawler distribuit scris în Go, conceput pentru extragerea de date cu concurență ridicată. Acesta funcționează ca un orchestrator de crawling distribuit și un motor de extragere dinamică a datelor, utilizând o arhitectură server-client pentru a coordona sarcinile pe mai multe noduri. Sistemul integrează un motor de browser headless pentru a randa conținut dinamic și a executa JavaScript, permițându-i să extragă date din aplicații single-page. Dispune de o interfață de gestionare bazată pe web pentru configurarea parametrilor spider-ului și monitorizarea progresului execuției, alături de capacitatea de a actualiza regulile de extragere prin fișiere de configurare cu hot-reloading, fără a reporni sistemul. Gestionarea traficului este realizată prin rotația pool-ului de proxy-uri și randomizarea cererilor pentru a evita detectarea bot-urilor și limitările de rată. Framework-ul include recuperarea stării bazată pe puncte de control pentru a relua sarcinile după eșecuri și oferă adaptoare de stocare pluggable pentru a exporta datele extrase în baze de date, cozi de mesaje sau fișiere.

Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica