Awesome GitHub RepositoriesDistributed Crawl Coordination

Firecrawl is a web data extraction platform designed to convert unstructured web content into clean, LLM-ready formats like markdown or JSON. It functions as an autonomous web crawler and scraper, capable of mapping entire domains, performing recursive navigation, and executing complex data gathering tasks. By leveraging headless browser orchestration, the system handles dynamic, JavaScript-heavy pages to ensure comprehensive data capture. The platform distinguishes itself through its focus on agentic workflows, providing a programmatic interface that allows autonomous agents to perform live

This project is a comprehensive educational guide and framework for building web scrapers using Python. It provides a course-based approach to data extraction, combining a Python crawler framework with tutorials on web reverse engineering and network traffic analysis. The project distinguishes itself by covering advanced extraction challenges, including the decryption of obfuscated JavaScript and the bypass of anti-scraping measures. It specifically addresses mobile application scraping through the simulation of user interactions and the interception of network traffic. The capability surfac

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Pholcus 是一个用 Go 编写的分布式网络爬虫框架，专为高并发数据提取而设计。它作为一个分布式爬虫编排器和动态数据提取引擎，利用服务器-客户端架构在多个节点间协调任务。 该系统集成了无头浏览器引擎来渲染动态内容并执行 JavaScript，从而能够从单页应用 (SPA) 中提取数据。它具有用于配置爬虫参数和监控执行进度的 Web 管理界面，并支持通过热重载配置文件更新提取规则，无需重启系统。 流量管理通过代理池轮换和请求随机化来处理，以规避机器人检测并避免速率限制。该框架包括基于状态的检查点恢复功能，可在故障后恢复任务，并提供可插拔的存储适配器，将提取的数据导出到数据库、消息队列或文件中。

Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica