Awesome GitHub RepositoriesJavaScript Crawling Frameworks

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

X-Ray 是一个 Web 抓取框架和异步 Web 爬虫，旨在从网站中提取结构化数据。它作为一个 HTML 数据提取器，使用 CSS 样式选择器将原始页面内容转换为定义的模式。 该项目实现了一个能够执行 JavaScript 以渲染动态内容的无头浏览器爬虫。它通过广度优先爬取策略和自动分页发现来处理网站内容发现，以遍历多页结果集。 该框架使用并发限制的请求队列和请求速率控制来管理 Web 数据管线，以调节传出的网络调用。提取的结果通过基于流的数据持久化进行处理，以在不占用系统内存的情况下处理大数据集。

Naabu is a port scanner library and tool that probes hosts for open ports using SYN, CONNECT, and UDP methods to identify active services. It functions as a Go library for embedding port scanning into programs, and as a standalone tool that accepts targets as hostnames, IP addresses, CIDR ranges, or ASN numbers. The tool discovers live hosts before scanning, filters ports by range or top lists, and can integrate with Nmap for service version detection. The project distinguishes itself through its SYN-based port probing approach that sends TCP SYN packets and analyzes responses without complet

这是一个分布式无头 Chrome 网络爬虫和数据提取框架。它作为一个 JavaScript 渲染引擎，使用无头浏览器处理动态页面，从需要 JavaScript 执行的网站中提取结构化数据。 该系统专为跨多个节点的分布式数据收集而设计，使用分布式任务同步和共享缓存来防止重复工作。它的特点是能够通过配置用户代理和视口尺寸来模拟特定的客户端环境，同时捕获页面截图等视觉证据。 该框架涵盖了全面的爬取管理，包括优先级队列请求调度、深度优先和广度优先遍历，以及对 robots.txt 和 sitemap.xml 文件的遵循。它提供了用于并发限制、事件监控以及将提取的数据流式传输到 CSV 或 JSON 格式的工具。

Hakrawler is a command-line web spider tool designed for security reconnaissance, built to crawl target websites and extract hyperlinks along with JavaScript file references. As a focused reconnaissance utility, it collects every discoverable URL and script source from a given domain, mapping the attack surface for penetration testing and vulnerability assessment. The tool differentiates itself through its concurrent architecture: a fixed-size goroutine pool fetches pages in parallel, while CSS selectors parse HTML to extract anchor and script references. A depth-aware recursion limiter preve

LinkFinder 是一个安全侦察和静态分析工具，专为 JavaScript 端点发现而设计。它从 JavaScript 文件中提取绝对和相对 URL 及参数，以映射 Web 应用程序的攻击面并识别隐藏的 API 路由。 该工具通过静态代码分析和正则表达式模式匹配来运行，无需执行源代码即可查找端点。它包括一个用于导入 Burp Suite 导出文件的数据处理器，从而能够在单次执行中对多个 JavaScript 资源进行批量分析。 该系统提供针对全域分析和特定域过滤的能力，以将发现重点放在目标上。它还具有关键字检测通知功能，可在结果中出现特定字符串时提醒用户，并支持将发现的数据导出为纯文本或 HTML 格式。

该项目是一个 Node.js Web 抓取框架，旨在通过请求、解析和文档交互的程序化工作流自动化数据提取。它作为一个无头 Web 爬虫、HTTP 请求管理器以及 DOM 解析器和提取器。 该框架通过结合用于与动态内容交互的 JavaScript 执行引擎和利用 CSS 及 XPath 选择器的混合选择系统脱颖而出。它包括用于代理轮换和 Cookie 罐会话管理的专用中间件，以维护身份验证状态并管理自动化流量。 其更广泛的功能涵盖递归链接爬取、分页处理和 Web 表单自动化。该工具还提供流量管理功能，例如通过定时延迟进行请求速率限制和自定义 HTTP 标头配置。

scrape-it 是一个 Node.js 网络爬虫和 HTML 解析器，旨在从网站和 HTML 文件中提取结构化数据。它作为一个网络数据提取工具，从 DOM 元素中检索特定信息，并将 Web 内容转换为可用的数据字段。 该工具使用 CSS 选择器定位特定数据点，并采用基于模式的数据映射将非结构化 Web 文本组织成一致的格式。它支持自定义值转换，将原始提取的字符串转换为特定的数据格式。 该系统提供了网络数据提取和自动化内容映射的功能。它可以解析来自 URL、原始 HTML 字符串或本地文件系统存储的 HTML 内容，并与无头浏览器集成以处理来自动态网页的内容。

Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data. The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic

simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.

Web Scraper is a chrome browser extension built for data extraction from web pages. Using this extension you can create a plan (sitemap) how a web site should be traversed and what should be extracted. Using these sitemaps the Web Scraper will navigate the site accordingly and extract all data.…

Webster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages.

Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.

js-crawler

Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head.