How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica
This project is a distributed headless Chrome web crawler and data extraction framework. It functions as a JavaScript rendering engine that uses a headless browser to process dynamic pages, extracting structured data from websites that require JavaScript execution. The system is designed for scalable data collection across multiple nodes, using distributed task synchronization and shared caches to prevent duplicate work. It distinguishes itself through the ability to emulate specific client environments by configuring user agents and viewport dimensions, while capturing visual evidence such a
Distribute crawler is a distributed web scraping framework that integrates with Scrapy to coordinate multiple crawler instances across clusters. It utilizes a centralized task queue to manage and scale concurrent data collection operations, enabling horizontal scaling of scraping tasks across multiple worker nodes. The framework distinguishes itself through its focus on large-scale data management and traffic control. It persists scraped items and binary assets into document-oriented database clusters, utilizing deduplication logic to optimize bandwidth and storage. To maintain consistent dat
Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a distributed spider engine and a headless browser renderer to execute JavaScript and extract content from dynamic web pages. The system includes a scalable data deduplicator to filter duplicate URLs and records during large-scale operations. A crawler monitoring system tracks the health of active scraping jobs and triggers alerts when system anomalies occur. The framework provides capabilities for task scheduling, web data extraction, and resilient workflows that allow crawling tas
This project is a distributed web crawling framework that enables the horizontal scaling of scraping tasks. It uses Redis as a centralized request queue manager and state store to coordinate crawl progress and request metadata across multiple server instances.
The main features of rolando/scrapy-redis are: Crawl Request Queues, Distributed Crawl State Stores, Distributed Web Crawling, Horizontal Crawler Scaling, Visited URL Sets, Distributed Crawl Workload Distribution, URL Duplicate Filters, Crawl Request Deduplications.
Projects with overlapping indexed features include: rmax/scrapy-redis — Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory… yujiosaka/headless-chrome-crawler — This project is a distributed headless Chrome web crawler and data extraction framework. It functions as a JavaScript… gnemoug/distribute_crawler — Distribute crawler is a distributed web scraping framework that integrates with Scrapy to coordinate multiple crawler… boris-code/feapder — Feapder is a Python web crawling framework designed for building scalable data extraction systems. It features a… binux/pyspider — PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for… scrapinghub/portia — Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as…