node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst
Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.
simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head.
Las características principales de n0tan3rd/squidwarc son: JavaScript Crawling Frameworks, Web Scraping.
Las alternativas de código abierto para n0tan3rd/squidwarc incluyen: cgiffard/node-simplecrawler — simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to… ionicabizau/scrape-it — scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files.… antivanov/js-crawler — js-crawler. brendonboshell/supercrawler — Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… lapwinglabs/x-ray — X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It…
