rchipka/node-osmosis

Features

• Web Crawlers - Functions as a programmable web crawler for asynchronously visiting URLs and extracting structured data.
• Web Scraping and Extraction - Implements a framework for parsing HTML and using selectors to extract structured data from websites.
• Web Scraping Frameworks - Offers a comprehensive framework for automating data extraction from websites via programmatic workflows.
• DOM Tree Construction - Converts raw HTML and XML strings into a hierarchical document object model for programmatic extraction.
• CSS and XPath Query Engines - Provides a query engine that locates page elements using both CSS3 selectors and XPath 1.0 expressions.
• HTML Selector Extractors - Implements CSS and XPath selectors to extract structured data from HTML and XML documents.
• CSS Selector - Extracts specific fields from HTML pages by evaluating CSS selectors against the DOM.
• HTTP Request Management - Provides comprehensive tools for executing and managing the lifecycle of HTTP requests including custom headers and proxies.
• XML and HTML Document Parsers - Parses HTML and XML content from strings into searchable in-memory document trees.
• Cookie-Based Session Management - Manages authenticated sessions by storing and matching cookies with domain and path specificity.
• Hybrid CSS-XPath Selectors - Retrieves specific elements and attributes using a hybrid of CSS and XPath syntax.
• HTTP Request Managers - Manages HTTP communication parameters, including cookies, custom headers, and proxy rotation.
• Web Crawling - Systematically discovers and navigates web content across domains for large-scale data collection.
• Domain-Restricted Link Navigation - Navigates through linked pages with configurable restrictions on internal and external domains.
• JavaScript Content Retrievals - Retrieves content from pages where data is generated dynamically via JavaScript by simulating events.
• Web Page Pagination Discovery - Automatically identifies and follows pagination links to traverse multi-page result sets.
• Recursive Web Discovery - Automatically discovers new pages by following hyperlinks recursively through website hierarchies.
• Custom Request Headers - Allows the definition of custom key-value pairs in outgoing HTTP request headers.
• Proxy Request Routers - Distributes outgoing requests across a pool of proxies to mask the origin IP and avoid bans.
• Proxy and User-Agent Rotation Middleware - Ships middleware that rotates proxies and user-agent strings to avoid detection during request processing.
• Proxy Routing - Distributes outgoing requests across multiple proxy servers to avoid IP-based rate limits.
• DOM Event Execution Environments - Provides an environment to run scripts that interact with dynamic content and trigger DOM events.
• Element-Targeted JavaScript Execution - Runs JavaScript within the document to interact with dynamic content and trigger DOM events.
• Randomized Request Delays - Inserts configurable pauses between requests to mimic human browsing patterns and avoid server rate limits.
• Request Rate Limiting - Introduces timed delays between requests to prevent server overload and evade detection.
• Dynamic Element Event Handling - Handles event listeners and scripts to interact with elements added to the DOM asynchronously.
• Programmatic Form Submissions - Triggers web form submissions programmatically by simulating button clicks and handling attributes.
• Dynamic Content Extraction - Extracts data from JavaScript-heavy pages by simulating browser events and rendering dynamic content.
• Contextual - Requests content using URLs dynamically constructed from request info, data objects, or context searches.
• Web Form Filling Tools - Provides tools to programmatically fill and submit web forms to access gated content.
• JavaScript Crawling Frameworks - HTML/XML parser and scraper for Node.js.
• Web Scraping - HTML/XML parser and web scraper.

Star history