ruipgil/scraperjs
Scraperjs
Scraperjs is a JavaScript web scraping library and headless browser automation tool designed to extract structured data from both static HTML and JavaScript-rendered websites. It functions as an asynchronous scraping workflow orchestrator and a selector-based engine for transforming unstructured web content into usable data.
The library distinguishes itself by managing a shared browser instance across multiple tasks to optimize resource usage and reduce startup latency. It supports custom script injection into sandboxed browser environments to manipulate the DOM or execute browser-side logic before data extraction.
The tool covers a broad range of capabilities, including dynamic and static content extraction, multi-site scraping workflows via URL pattern mapping, and the coordination of HTTP requests through promise-based lifecycle chains.
Features
- Automated Web Data Collection - Converts unstructured web content into structured data through programmable sequences of requests and parsing.
- Headless Browser Automation - Provides a framework for programmatically controlling headless browsers to render dynamic content and extract data.
- Automated Web Scraping - Automates the collection of structured information from websites using an asynchronous workflow.
- Content Extraction - Parses static HTML using selectors to retrieve specific elements and attributes.
- DOM-Based Extractions - Transforms unstructured web content into usable data by reading DOM elements via selector-based extraction.
- Structured Web Data Parsing - Parses content from both static and dynamic web pages to collect structured information.
- Web Data Scraping - Extracts data from JavaScript-heavy websites by rendering pages in a headless browser.
- Live Page Content Extraction - Retrieves elements and attributes from live or static web pages using a selector-based interface.
- Browser Automation - Provides programmatic control of browsers to manipulate the DOM and extract information.
- Scraping Workflow Orchestrators - Acts as a promise-based manager for sequencing HTTP requests, timeouts, and URL routing during data collection.
- DOM Selectors - Uses CSS and XPath selectors to extract structured data and attributes from HTML elements.
- Dynamic Content Extraction - Renders JavaScript-heavy websites in a headless browser to retrieve content that requires client-side execution.
- Headless Browser Orchestration - Provides a system for managing headless browser instances to render dynamic content and execute JavaScript across multiple tasks.
- Web Scraping and Extraction - Implements a programmable interface for parsing HTML and extracting structured data from both static and dynamic websites.
- Multi-Site Scraping Workflows - Maps different URL patterns to unique extraction logic to traverse and scrape multiple website structures simultaneously.
- Regex-Based Route Matching - Uses regular expressions to match URL patterns and apply specific scraping logic to different website structures.
- URL Pattern Matching - Maps specific URL patterns to unique scraping logic to traverse complex website structures.
- Promise-Based Chains - Implements asynchronous promise chains to sequence HTTP requests and data extraction tasks.
- Browser Instance Pooling - Optimizes resource usage by sharing a single headless browser instance across multiple scraping sessions.
- Scraper Lifecycles - Orchestrates sequences of requests and extraction tasks using a promise chain with integrated timeout and status triggers.
- Instance Pooling - Optimizes resource usage by sharing a single headless browser instance across multiple scraping tasks.
- Script Injections - Allows the injection of custom JavaScript into the browser environment to manipulate the DOM before extraction.
- JavaScript Crawling Frameworks - Versatile web scraper for Node.js.
- Web Scraping - Versatile web scraping utility.
