rmax/scrapy-redis
Scrapy Redis
Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress.
The library provides a Redis-based duplicate filter that deduplicates URLs across distributed crawls, and a Redis item pipeline that pushes scraped items into a Redis queue for asynchronous consumption by downstream processors. It supports dynamic injection of crawl targets by reading structured JSON payloads from Redis and converting them into HTTP requests with associated metadata and cookies, allowing external processes to feed new URLs into the shared queue while spiders consume them in real time. Configuration is handled through standard Redis connection parameters such as host, port, or connection URL.
The project covers the full lifecycle of distributed web crawling: request scheduling and distribution, URL deduplication, crawl state persistence, and scraped item queuing. It replaces Scrapy's built-in scheduler, dupefilter, and item pipeline with Redis-backed equivalents, making it straightforward to scale crawling across multiple machines.
Features
- Distributed Crawl Coordination - Coordinates multiple Scrapy spiders that share a Redis-backed request queue and deduplicate URLs across workers.
- Redis-Backed Schedulers - Replaces Scrapy's in-memory scheduler with a Redis-backed one that coordinates request distribution across workers.
- Redis - Pushes scraped data into Redis queues for downstream batch processing or consumption by separate services.
- Web Crawl Schedulers - Implements a distributed scheduler that coordinates crawl request distribution across multiple Scrapy spider workers via Redis.
- Scraped Item Sinks - Pushes scraped items into a Redis list for asynchronous consumption by downstream processors.
- Scraped Data Storage - Persists each scraped item into a Redis list for later batch processing or consumption by other services.
- Redis Item Queues - Pushes scraped items into a Redis queue so separate processes can consume and process them independently.
- Visited URL Sets - Uses a Redis set to track seen URLs across all workers, preventing duplicate crawls.
- Distributed Job Execution - Shares a Redis-backed request queue among multiple spider instances so each worker picks the next unprocessed URL.
- Crawl Request Schedulers - Pushes new URLs into a Redis list and lets any connected spider consume them, enabling dynamic feed-in of crawl targets.
- Crawl State Recovery - Saves crawled URLs and pending requests in Redis to survive restarts and enable resume.
- Redis-Backed Queues - Stores pending crawl requests in Redis lists so multiple spiders can consume them concurrently.
- Crawl Request Injectors - Reads JSON payloads from Redis and converts them into structured HTTP requests with metadata and cookies.
- Crawl Request Queues - Provides a Redis-backed queue that stores and distributes HTTP crawl requests across multiple Scrapy spider workers.
- URL Duplicate Filters - Ships a Redis-based duplicate filter that prevents the same URL from being crawled twice across distributed spider workers.
- Crawl Request Injections - Reads JSON payloads from Redis and converts them into structured HTTP requests with metadata and cookies.
- Crawl Request Deduplications - Uses a Redis set to filter duplicate URLs across all running spiders, preventing the same page from being crawled twice.
- Dynamic URL Injections - Feeds new crawl targets into a shared Redis queue from external processes while spiders consume them in real time.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
