rmax/scrapy-redis
Scrapy Redis
Scrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress.
The library provides a Redis-based duplicate filter that deduplicates URLs across distributed crawls, and a Redis item pipeline that pushes scraped items into a Redis queue for asynchronous consumption by downstream processors. It supports dynamic injection of crawl targets by reading structured JSON payloads from Redis and converting them into HTTP requests with associated metadata and cookies, allowing external processes to feed new URLs into the shared queue while spiders consume them in real time. Configuration is handled through standard Redis connection parameters such as host, port, or connection URL.
The project covers the full lifecycle of distributed web crawling: request scheduling and distribution, URL deduplication, crawl state persistence, and scraped item queuing. It replaces Scrapy's built-in scheduler, dupefilter, and item pipeline with Redis-backed equivalents, making it straightforward to scale crawling across multiple machines.
