bda-research/node-crawler

Features

• Web Crawling - Queues and visits large sets of URLs asynchronously while managing request retries and preventing duplicate processing.
• Web Crawlers - Provides a programmable Node.js framework for managing request queues and automating data extraction.
• HTML Parsing - Extracts data from HTML responses using a server-side DOM implementation and CSS-style selectors.
• Web Data Extraction - Implements programmatic scraping and processing of web content to extract structured data.
• Asynchronous Crawl Queues - Manages asynchronous crawl queues for long-running data extraction jobs.
• HTML Parsers - Includes a headless parser that converts HTML responses into a DOM structure for data extraction.
• High Performance Scraping - Uses HTTP/2 and concurrent connection management to collect data quickly while respecting target server load limits.
• Multi-Protocol Handlers - Supports both HTTP/1.1 and HTTP/2 protocols to optimize connection performance and ensure server compatibility.
• HTTP/2 Support - Implements HTTP/2 support to enhance request performance and ensure compatibility with modern servers during large-scale scraping.
• Proxy Rotation Services - Distributes network traffic across a pool of proxy servers to bypass rate limits.
• Proxy and Fingerprint Rotation - Distributes outgoing network traffic across a pool of proxy servers and user agents to bypass access restrictions.
• User Agent Rotation - Cycles through different user agent strings to mimic various browser environments and avoid detection.
• Asynchronous Task Queues - Manages a list of URLs for asynchronous retrieval while preventing duplicate entries through task deduplication.
• Request Rate Limiting - Limits the frequency of outgoing requests and concurrent connections to prevent server overload.
• ID-Based Rate Limiting - Regulates request frequency and concurrency by grouping outgoing traffic into independent buckets mapped to specific target IDs.
• Rate-Limited Clients - Ships a request client with built-in concurrency controls and per-target rate limiting.
• XML Parsing - Processes page content by recognizing MIME types and using DOM manipulation to extract data from XML.
• Request Retries - Automatically attempts to refetch failed pages to increase the success rate of data collection.
• JSON Response Parsers - Includes utilities to treat response bodies as JSON and disable HTML parsing for simplified API data extraction.
• Pre-Request Logic Hooks - Executes custom synchronous or asynchronous functions before each queued request to modify options or prepare state.
• Crawlers - Utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping.
• MIME-Aware Content Parsers - Detects response types to switch between raw binary downloads, JSON extraction, or DOM-based HTML and XML processing.
• Remote File Downloads - Retrieves raw response bodies without string conversion to save images, PDFs, and other non-text assets.
• Request Header Configuration - Allows the configuration of custom HTTP headers globally or per request to manage client identification.
• Crawler Lifecycle Hooks - Provides event-driven hooks to trigger custom logic during task scheduling, request dispatch, and queue drainage.
• URL Request Tracking - Tracks visited URLs in a local registry to prevent redundant network requests.
• Target-Based - Groups requests by ID so that different targets or proxies maintain distinct rate limits and connection caps.
• Encoding Normalizers - Analyzes response headers and meta tags to normalize diverse character sets into a standard UTF-8 format.
• Duplicate Prevention - Prevents the same URL or task from entering the queue multiple times to avoid redundant processing.
• JavaScript Crawling Frameworks - Simple API-driven crawler for Node.js.
• Web Scraping - Web crawler with jQuery-like parsing.

Star history