3 repository-uri
Awesome GitHub RepositoriesScraped Data Storage
Storage solutions specifically tailored for the high-volume persistence of scraped web data.
Distinct from Analytics Data Stores: Existing candidates focus on in-memory stores or analytics metrics rather than general scraped content persistence.
Explore 3 awesome GitHub repositories matching data & databases · Scraped Data Storage. Refine with filters or upvote what's useful.
Awesome Scraped Data Storage GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- rmax/scrapy-redis
rmax/scrapy-redis
5,639Vezi pe GitHubScrapy-Redis is a library that transforms Scrapy into a distributed web crawling framework by replacing its in-memory scheduler with a Redis-backed component. This allows multiple Scrapy spider workers to coordinate through a shared request queue, enabling them to consume URLs concurrently while a Redis set tracks seen URLs across all workers to prevent duplicate crawls. The system persists crawl state—including pending requests and already-crawled URLs—in Redis, so a paused or crashed spider can resume from where it left off without losing progress. The library provides a Redis-based duplica
Pushes scraped items into a Redis queue so separate processes can consume and process them independently.
Pythoncrawlerdistributedredis - rolando/scrapy-redis
rolando/scrapy-redis
5,639Vezi pe GitHubAcest proiect este un framework distribuit de web crawling care permite scalarea orizontală a sarcinilor de scraping. Utilizează Redis ca manager centralizat de cozi de cereri și stocare de stare pentru a coordona progresul crawl-ului și metadatele cererilor pe mai multe instanțe de server. Sistemul distribuie sarcinile de crawling prin partajarea unei singure cozi de cereri și utilizează un filtru distribuit de duplicate pentru a preveni vizitarea aceleiași pagini de către mai mulți lucrători. Acesta persistă starea complexă a cererilor și metadatele sub formă de șiruri JSON în cadrul stocării remote partajate. Framework-ul oferă, de asemenea, capabilități pentru procesarea distribuită a datelor prin trimiterea elementelor extrase într-o coadă partajată pentru consumul paralel de către lucrători de procesare separați.
Pushes scraped items into a shared Redis queue for parallel consumption by separate processing workers.
Python - kr1s77/python-crawler-tutorial-starts-from-zero
Kr1s77/Python-crawler-tutorial-starts-from-zero
4,599Vezi pe GitHubThis project is a Python web scraping tutorial and framework designed for building automated data extraction tools and web crawlers. It provides a structured approach to navigating websites and persisting scraped data to databases. The project includes a toolset for web API analysis, focusing on reverse engineering obfuscated API requests and inspecting network traffic to extract structured data. It also covers optical character recognition workflows to convert visual text within images into machine-readable strings. The framework covers capabilities for headless browser automation to handle
Enables the storage of large volumes of unstructured scraped data in databases for future analysis.
Python
