How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
simplecrawler is designed to provide a basic, flexible and robust API for crawling websites. It was written to archive, analyse, and search some very large websites and has happily chewed through hundreds of thousands of pages and written tens of gigabytes to disk without issue.
The main features of cgiffard/node-simplecrawler are: JavaScript Crawling Frameworks, Web Scraping.
Projects with overlapping indexed features include: ionicabizau/scrape-it — scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files.… lapwinglabs/x-ray — X-Ray is a web scraping framework and asynchronous web crawler designed to extract structured data from websites. It… antivanov/js-crawler — js-crawler. brendonboshell/supercrawler — Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. bda-research/node-crawler — node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It… martinsbalodis/web-scraper-chrome-extension — Web Scraper is a chrome browser extension built for data extraction from web pages. Using this extension you can…
node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst
Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use.
scrape-it is a Node.js web scraper and HTML parser designed to extract structured data from websites and HTML files. It functions as a web data extraction tool that retrieves specific information from DOM elements and converts web content into usable data fields. The tool uses CSS selectors to target specific data points and employs schema-driven data mapping to organize unstructured web text into a consistent format. It supports custom value transformation to convert raw extracted strings into specific data formats. The system provides capabilities for web data extraction and automated cont
