rolando/scrapy-redis
Scrapy Redis
Este proyecto es un framework de rastreo web distribuido que permite el escalado horizontal de tareas de scraping. Utiliza Redis como gestor de colas de solicitudes centralizado y almacén de estado para coordinar el progreso del rastreo y los metadatos de las solicitudes a través de múltiples instancias de servidor.
El sistema distribuye las cargas de trabajo de rastreo compartiendo una única cola de solicitudes y utiliza un filtro de duplicados distribuido para evitar que múltiples trabajadores visiten la misma página. Persiste el estado complejo de la solicitud y los metadatos como cadenas JSON dentro del almacén remoto compartido.
El framework también proporciona capacidades para el procesamiento de datos distribuido al enviar elementos scrapeados a una cola compartida para el consumo paralelo por parte de trabajadores de procesamiento separados.
Features
- Crawl Request Queues - Provides a centralized Redis-backed queue to coordinate HTTP crawl requests across distributed worker instances.
- Distributed Crawl State Stores - Provides a shared Redis storage backend for tracking visited URLs and crawl progress across server instances.
- Distributed Web Crawling - Provides a framework for scaling Scrapy crawlers across multiple servers using Redis for coordination.
- Horizontal Crawler Scaling - Increases crawling throughput by running multiple Scrapy instances pulling from a single centralized Redis queue.
- Visited URL Sets - Implements shared visited URL sets in Redis to ensure collaborative deduplication across a crawler cluster.
- Distributed Crawl Workload Distribution - Distributes crawling tasks across several server instances by sharing a single centralized request queue.
- URL Duplicate Filters - Implements a shared Redis set to prevent multiple distributed workers from crawling the same URL.
- Crawl Request Deduplications - Prevents redundant crawling of the same page by tracking visited URLs in a shared Redis set.
- Scrapy-Framework-Based Crawlers - Extends the Scrapy framework to support distributed crawling using Redis for queue and deduplication coordination.
- Web Crawling Frameworks - Serves as a comprehensive framework for scaling web scraping tasks horizontally via shared state.
- Distributed Data Processing - Facilitates distributed data processing by pushing scraped items into shared queues for parallel worker consumption.
- Redis Item Queues - Pushes scraped items into a shared Redis queue for parallel consumption by separate processing workers.
- Python Crawling Frameworks - Redis-based components for distributed Scrapy projects.
