3 Repos
Awesome GitHub RepositoriesConcurrent Scraping Workers
Parallel execution systems for high-volume web data extraction across multiple browser sessions.
Distinct from Concurrent URL Processors: Unlike the candidates, this focuses on scaling headless browser sessions for data extraction rather than network packet processing or database queries.
Explore 3 awesome GitHub repositories matching data & databases · Concurrent Scraping Workers. Refine with filters or upvote what's useful.
Awesome Concurrent Scraping Workers GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- h4ckf0r0day/obscura
h4ckf0r0day/obscura
16,110Auf GitHub ansehenObscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for AI models to control browser sessions, interact with websites, and extract web data using a WebSocket implementation of the Chrome DevTools Protocol. The project focuses on bot detection evasion by randomizing browser fingerprints, masking native functions, and blocking tracking scripts to mimic human behavior. It further secures identities through a traffic layer that routes network requests via HTTP or SOCKS5 proxies. The system supports large-scale data extraction through
Implements parallel worker execution to efficiently extract large volumes of data from multiple URLs.
Rustantidetectantidetect-browserbrowser - tophubs/toplist
tophubs/TopList
4,730Auf GitHub ansehenTopList ist ein Aggregator für aktuelle Nachrichten und ein Dashboard zur Schlagzeilen-Aggregation. Als Go-Web-Crawler konzipiert, fungiert es als zentrales Tool zum Sammeln und Anzeigen populärer aktueller Themen aus verschiedenen Online-Nachrichten- und sozialen Quellen in einer einzigen Ansicht. Das System konzentriert sich auf das Multi-Source-Content-Scraping und die Überwachung von Web-Schlagzeilen. Es ruft Informationen von verschiedenen Webseiten ab, um trendige Schlagzeilen in einem einheitlichen Feed zu konsolidieren.
Employs concurrent Go routines to execute high-volume web data extraction across multiple remote target websites.
Gogolanghothotlist - dotnetcore/dotnetspider
dotnetcore/DotnetSpider
4,137Auf GitHub ansehenDotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites. The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem. The system utilizes a pipeline-based data
Provides concurrent scraping workers to maximize throughput during large-scale web data extraction.
C#crawlercross-platformcsharp
