3 repositorios
Awesome GitHub RepositoriesScraped Data Storage
Storage solutions specifically tailored for the high-volume persistence of scraped web data.
Distinct from Analytics Data Stores: Existing candidates focus on in-memory stores or analytics metrics rather than general scraped content persistence.
Explore 3 awesome GitHub repositories matching data & databases · Scraped Data Storage. Refine with filters or upvote what's useful.
Awesome Scraped Data Storage GitHub Repositories
Encuentra los mejores repositorios con IA.Buscaremos los repositorios que mejor coincidan usando IA.
- rmax/scrapy-redis
rmax/scrapy-redis
5,639Ver en GitHubScrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica
Pushes scraped items into a Redis queue so separate processes can consume and process them independently.
Pythoncrawlerdistributedredis - rolando/scrapy-redis
rolando/scrapy-redis
5,639Ver en GitHubEste proyecto es un framework de rastreo web distribuido que permite el escalado horizontal de tareas de scraping. Utiliza Redis como gestor de colas de solicitudes centralizado y almacén de estado para coordinar el progreso del rastreo y los metadatos de las solicitudes a través de múltiples instancias de servidor. El sistema distribuye las cargas de trabajo de rastreo compartiendo una única cola de solicitudes y utiliza un filtro de duplicados distribuido para evitar que múltiples trabajadores visiten la misma página. Persiste el estado complejo de la solicitud y los metadatos como cadenas JSON dentro del almacén remoto compartido. El framework también proporciona capacidades para el procesamiento de datos distribuido al enviar elementos scrapeados a una cola compartida para el consumo paralelo por parte de trabajadores de procesamiento separados.
Pushes scraped items into a shared Redis queue for parallel consumption by separate processing workers.
Python - kr1s77/python-crawler-tutorial-starts-from-zero
Kr1s77/Python-crawler-tutorial-starts-from-zero
4,599Ver en GitHubThis project is a Python web scraping tutorial and framework designed for building automated data extraction tools and web crawlers. It provides a structured approach to navigating websites and persisting scraped data to databases. The project includes a toolset for web API analysis, focusing on reverse engineering obfuscated API requests and inspecting network traffic to extract structured data. It also covers optical character recognition workflows to convert visual text within images into machine-readable strings. The framework covers capabilities for headless browser automation to handle
Enables the storage of large volumes of unstructured scraped data in databases for future analysis.
Python
