Awesome GitHub RepositoriesData Extraction

Changedetection.io is a self-hosted monitoring service designed to track web pages for content updates and notify users of changes. It functions as a centralized platform where users can manage tracking tasks, observe specific website elements, and receive automated alerts through various communication channels whenever modifications are detected. The service distinguishes itself through an integrated headless browser engine that executes interaction sequences, such as logins or form submissions, to access dynamic or restricted content. It maintains a historical record of page snapshots, util

Katana is a web crawler and spider designed for security reconnaissance and web application mapping. It functions as a utility for identifying endpoints, forms, and API structures across web targets by combining standard HTTP request traversal with headless browser automation to render dynamic, JavaScript-heavy content. The tool distinguishes itself through its ability to maintain authenticated sessions and handle complex web interactions, such as automated form submission and captcha resolution. It provides granular control over the discovery process, allowing users to define specific crawl

Photon is a command-line web crawler designed for security reconnaissance and information gathering. It systematically traverses websites to discover URLs, map domain infrastructure, and identify associated subdomains by retrieving DNS records. The tool distinguishes itself through its ability to perform deep content analysis, including the extraction of sensitive data such as API keys and authentication tokens using user-defined regular expressions. It supports offline inspection by cloning crawled web content to the local filesystem, allowing for structural analysis without additional netwo

Firefox is a cross-platform web browser engine designed to render web content, execute JavaScript, and manage secure browsing sessions. It utilizes a multi-process isolation architecture that distributes browser tasks across independent operating system processes to ensure stability and prevent site-specific failures from impacting the entire application. The engine incorporates a sandboxed execution environment to restrict web content and untrusted scripts to isolated memory compartments, enforcing security policies that prevent unauthorized access to system resources. The project distinguis

weiboSpider is a Python web scraper and social media crawler designed to extract user profiles, posts, and engagement metrics from Sina Weibo. It functions as an automated data pipeline for academic research and trend analysis, collecting long-form text and multimedia content. The tool distinguishes itself through the use of browser session cookies to authenticate requests and access protected profiles. It implements randomized request pacing and global pauses to manage traffic and avoid platform rate limits, while supporting incremental crawling to capture only new content based on timestamp

Pholcus 是一个用 Go 编写的分布式网络爬虫框架，专为高并发数据提取而设计。它作为一个分布式爬虫编排器和动态数据提取引擎，利用服务器-客户端架构在多个节点间协调任务。 该系统集成了无头浏览器引擎来渲染动态内容并执行 JavaScript，从而能够从单页应用 (SPA) 中提取数据。它具有用于配置爬虫参数和监控执行进度的 Web 管理界面，并支持通过热重载配置文件更新提取规则，无需重启系统。 流量管理通过代理池轮换和请求随机化来处理，以规避机器人检测并避免速率限制。该框架包括基于状态的检查点恢复功能，可在故障后恢复任务，并提供可插拔的存储适配器，将提取的数据导出到数据库、消息队列或文件中。

pydoll is a Chrome DevTools Protocol automation library and headless browser controller used for web data extraction and parallel browser automation. It controls Chromium-based browsers via direct WebSocket connections, allowing it to manage isolated browser contexts and tabs while bypassing the overhead and detection associated with WebDriver. The project features an anti-bot evasion framework that mimics natural human behavior, including mouse movements generated via Bezier curves and variable typing patterns. It provides specialized stealth capabilities to bypass behavioral analysis and au

Steel is a cloud browser automation platform that provides a REST API for launching and controlling remote Chrome browser sessions. It enables programmatic browsing and web scraping using standard automation tools like Puppeteer, Playwright, and Selenium, connecting to cloud-hosted browser instances via WebSocket and the Chrome DevTools Protocol. The platform supports both headless and headful browser sessions, with language-specific SDKs for TypeScript and Python. The service distinguishes itself through comprehensive anti-detection capabilities, including residential proxy rotation, CAPTCHA

Open Deep Research is an AI-powered web research agent that combines a reasoning model with live web search and data extraction to perform deep, multi-source investigations on any topic. It operates through a dual interface, offering both a command-line tool and a Model Context Protocol server, allowing developers to integrate web capabilities directly into AI agents and coding assistants. The project distinguishes itself by orchestrating an iterative research loop where a reasoning model plans steps, interprets search results, and guides subsequent web interactions. It uses Firecrawl for scr

X-Ray 是一个 Web 抓取框架和异步 Web 爬虫，旨在从网站中提取结构化数据。它作为一个 HTML 数据提取器，使用 CSS 样式选择器将原始页面内容转换为定义的模式。 该项目实现了一个能够执行 JavaScript 以渲染动态内容的无头浏览器爬虫。它通过广度优先爬取策略和自动分页发现来处理网站内容发现，以遍历多页结果集。 该框架使用并发限制的请求队列和请求速率控制来管理 Web 数据管线，以调节传出的网络调用。提取的结果通过基于流的数据持久化进行处理，以在不占用系统内存的情况下处理大数据集。

这是一个分布式无头 Chrome 网络爬虫和数据提取框架。它作为一个 JavaScript 渲染引擎，使用无头浏览器处理动态页面，从需要 JavaScript 执行的网站中提取结构化数据。 该系统专为跨多个节点的分布式数据收集而设计，使用分布式任务同步和共享缓存来防止重复工作。它的特点是能够通过配置用户代理和视口尺寸来模拟特定的客户端环境，同时捕获页面截图等视觉证据。 该框架涵盖了全面的爬取管理，包括优先级队列请求调度、深度优先和广度优先遍历，以及对 robots.txt 和 sitemap.xml 文件的遵循。它提供了用于并发限制、事件监控以及将提取的数据流式传输到 CSV 或 JSON 格式的工具。

这是一个新浪微博网页爬虫和社交媒体数据管道，旨在提取用户资料、帖子、评论和多媒体资源。它作为一个容器化的数据爬虫，自动化收集社交媒体内容和互动指标，并将其存储在本地。 该系统包含一个处理层，利用大语言模型分析抓取的文本，生成摘要和情感分析。它通过一个部署就绪的容器模型脱颖而出，该模型具有用于管理提取任务和监控作业进度的 HTTP 界面。 该爬虫涵盖了广泛的功能，包括通过定时增量更新进行社交媒体监控、将多媒体资源归档到本地磁盘，以及向平面文件或数据库进行多格式数据导出。它还能捕获详细的社交互动，如一级评论和转发。

CrawlerTutorial 是一个全面的 Python 网络爬虫教程和框架，旨在从静态和动态网站中提取数据。它作为一个网络数据提取管道和 HTTP 请求编排器，涵盖了从初始获取到最终数据存储的爬虫应用程序全生命周期。 该项目提供了关于反机器人绕过技术和 Web API 逆向工程的专业指导。它包括通过身份掩码和代理轮换规避浏览器检测的方法，以及通过分析网络流量和请求签名识别隐藏 API 端点的技术。 该框架包含广泛的功能，包括针对 JavaScript 重度页面的浏览器自动化、通过 QR 码或短信的自动用户身份验证以及会话持久性管理。它还具有用于清理原始文本、删除重复记录并将收集到的信息持久化到平面文件或关系数据库中的数据预处理工具。

Scylla 是一个用于管理 HTTP 代理池并实现网页抓取自动化的系统。它提供了一个专门的数据采集流水线，旨在为训练和微调大语言模型收集大规模互联网数据集。 该项目具有代理轮换网关，可为传入请求分配新的代理地址，以隐藏源流量并避免 IP 被封。它包含一个代理池管理器，负责代理服务器的收集、功能验证和编排，并配有一个用于监控网络健康状况和地理分布的 Web 仪表板。 该系统支持使用无头浏览器爬虫进行动态内容提取，以捕获依赖 JavaScript 渲染的网站数据。它还提供了一个用于代理管理和与外部抓取框架集成的 JSON 接口。

该项目是一个未经身份验证的 Web 爬虫，旨在从 Twitter 前端 API 提取公共数据。它作为一个社交媒体数据提取器，模拟浏览器请求以收集信息，而无需官方 API 密钥或用户账户身份验证。 该工具提供了收集公共帖子、抓取用户个人资料元数据（如简介和位置）以及检索按地理区域分类的热门话题的功能。它可以根据特定的用户名、标签或搜索查询执行有针对性的内容抓取。 该系统通过分页、基于标头的客户端模拟来模仿 Web 浏览器，并解析来自公共端点的结构化 JSON 响应来管理数据收集。