2 个仓库
Awesome GitHub RepositoriesDistributed Web
The use of distributed scraping architectures to collect high volumes of web data for analysis.
Distinct from Data Mining: Specific to web scraping throughput scaling, whereas general Data Mining focuses on pattern extraction from existing datasets.
Explore 2 awesome GitHub repositories matching data & databases · Distributed Web. Refine with filters or upvote what's useful.
Awesome Distributed Web GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- asciimoo/colly
asciimoo/colly
25,348在 GitHub 上查看Colly is a web scraping framework and concurrent crawler written in Go. It provides a system for traversing web pages, following links, and extracting structured data from HTML and XML documents. The framework includes a distributed scraping engine designed to spread data collection tasks across multiple instances to increase throughput. It ensures compliance with website owner policies by automatically reading and respecting robots.txt files. The system manages request lifecycles through domain-based rate limiting, concurrency controls, and session management via a stateful cookie jar. It s
Distributes scraping tasks across multiple instances to increase the volume and throughput of collected web data.
Go - zlzforever/dotnetspider
zlzforever/DotnetSpider
4,136在 GitHub 上查看DotnetSpider 是一个 .NET 网络爬虫框架和可编程工具，旨在遍历网站并从网页中捕获结构化数据。它作为一个分布式爬虫引擎，支持自动化网络爬虫以发现和提取数据。 该框架专为分布式数据提取而设计，允许将爬取任务分散到多台服务器上，以处理海量 Web 内容。这种架构支持高性能 Web 抓取和企业级数据收集工作流，用于收集结构化信息。
Uses a distributed scraping architecture to collect high volumes of web data for analysis.
C#
