Awesome GitHub RepositoriesWeb Crawling Frameworks

PySpider is a Python web crawling framework designed for automated data extraction. It provides a pipeline for periodically fetching web content, processing HTML, and persisting scraped information into database backends. The system features a web-based management interface for editing scraping scripts, monitoring task progress, and reviewing collected data. It includes a headless browser JavaScript renderer to capture rendered HTML from dynamic web pages and a distributed architecture that uses message queues to scale crawling workloads across multiple nodes. The framework also covers task

Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

Portia is a containerized scraping platform and visual web scraper that enables no-code data extraction. It serves as a Scrapy visual scraping tool and spider generator, allowing users to design and deploy web scrapers through a graphical interface instead of writing manual selector code. The system distinguishes itself by converting visual web page annotations into executable Scrapy spider code and structured JSON specifications. This visual-to-code mapping allows users to define scraping logic and extraction rules through a point-and-click interface, which can then be exported for use in ex

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

This project is a self-hosted system for discovering, browsing, and receiving personalized recommendations from academic papers on arXiv. It combines an arXiv API client that downloads paper metadata and PDFs with a TF-IDF document similarity engine and an SVM-based recommendation system that trains a classifier per user based on their preferences. The system provides a web interface for browsing, searching, and filtering recent arXiv submissions, alongside personalized paper recommendations generated from individual user signals. It also includes a Twitter mention tracker that periodically p

这是一个分布式网络爬取框架，支持抓取任务的水平扩展。它使用 Redis 作为集中式请求队列管理器和状态存储，以协调跨多个服务器实例的爬取进度和请求元数据。 该系统通过共享单个请求队列来分发爬取工作负载，并利用分布式重复过滤器来防止多个工作节点访问同一页面。它将复杂的请求状态和元数据作为 JSON 字符串持久化在共享远程存储中。 该框架还通过将抓取到的项目推送到共享队列以供单独的处理工作节点并行消费，从而提供分布式数据处理功能。

OpenChat is a conversational AI agent builder and customer service automation platform that uses large language models to power customer support chatbots across multiple channels. It provides tools for defining AI agent behavior, training on custom knowledge, managing actions, and controlling autopilot responses per channel. The platform enables deploying AI agents on web, phone, email, SMS, and WhatsApp, with a unified inbox for managing conversations across all channels. It includes CRM synchronization, automated workflows, contact segmentation, and analytics for tracking customer satisfact

Crawler4j 是一个多线程 Java 网络爬虫，专为高容量 Web 遍历和内容提取而设计。它作为一个“礼貌”的爬取框架，能够发现并索引多个网站上的 HTML 和二进制内容。 该项目通过一种持久化爬取模型脱颖而出，该模型将会话状态序列化到本地存储，允许引擎在崩溃或中断后恢复索引。它包含一个礼貌控制器来调节请求频率和延迟，防止服务器过载和 IP 被封。 该系统涵盖了广泛的遍历功能，包括深度限制范围管理、目标过滤，以及针对自定义用户代理和代理路由的请求拦截。数据存储通过存储库模式处理，将爬取逻辑与关系数据库中页面元数据的持久化解耦。

DotnetSpider 是一个 .NET 网络爬虫框架和 C# 数据提取工具，旨在实现大规模的自动化网页发现和互联网结构化数据检索。它作为一个高级 Web 抓取库，用于从各种网站收集信息。 该框架提供了自动化网络爬虫和大规模数据抓取的功能。它支持网页内容提取，以便在 .NET 生态系统内通过程序化 Web 自动化创建本地数据库或分析在线信息。 系统利用基于流水线的数据处理模型，具有异步请求处理和并发工作线程执行功能。它具有基于任务队列的调度器、模块化存储提供程序以及用于自定义抓取逻辑的接口驱动实现。

MNBVC is a dataset pipeline and toolkit designed for the collection, cleaning, and normalization of massive text and code corpora used to train large language models. It provides specialized tools for harvesting source code, commit histories, and repository metadata from version control platforms, alongside a multilingual text corpus collector for gathering parallel text and academic papers. The project distinguishes itself through comprehensive capabilities for processing diverse document types, including a PDF-to-text converter that transforms complex layouts and formulas into structured JS

X-crawl is a Node.js-based web scraping framework designed to automate data collection from both static and dynamic websites. It integrates artificial intelligence to perform semantic parsing, allowing it to transform unstructured HTML into structured data formats that remain accurate even when website layouts or class names change. The project distinguishes itself through a comprehensive suite of stealth and reliability features. It manages crawler identity by randomizing device fingerprints and rotating proxy servers to bypass access restrictions. To handle complex, JavaScript-heavy interfa