node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst
Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.
simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.
Webster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages.
zhuyingda/webster 的主要功能包括：JavaScript Crawling Frameworks, Web Scraping。
zhuyingda/webster 的开源替代品包括： cgiffard/node-simplecrawler — simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to… ionicabizau/scrape-it — scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files.… antivanov/js-crawler — js-crawler. brendonboshell/supercrawler — Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… lapwinglabs/x-ray — X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It…
