yujiosaka/headless-chrome-crawler
Headless Chrome Crawler
This project is a distributed headless Chrome web crawler and data extraction framework. It functions as a JavaScript rendering engine that uses a headless browser to process dynamic pages, extracting structured data from websites that require JavaScript execution.
The system is designed for scalable data collection across multiple nodes, using distributed task synchronization and shared caches to prevent duplicate work. It distinguishes itself through the ability to emulate specific client environments by configuring user agents and viewport dimensions, while capturing visual evidence such as page screenshots.
The framework covers comprehensive crawl management, including priority-queue request scheduling, depth-first and breadth-first traversal, and adherence to robots.txt and sitemap.xml files. It provides tools for concurrency limiting, event monitoring, and streaming extracted data into CSV or JSON formats.
Features
- Headless Rendering Engines - Provides a headless Chrome rendering engine to execute JavaScript and process dynamic page content.
- Website Crawlers and Scrapers - Implements a recursive website crawler and scraper that handles navigation and converts pages into structured formats.
- JavaScript Rendering - Provides retrieval of data from dynamic pages by executing JavaScript via headless browsers before parsing HTML.
- Distributed Cache Synchronizers - Uses distributed cache synchronizers to track requested URLs across multiple servers and prevent duplicate work.
- Distributed Web Crawling - Scales data collection across multiple nodes using coordinated request queues for large-scale web crawling.
- Crawl History Deduplication - Prevents duplicate network requests by storing a history of visited URLs in an external database.
- Structured Data Extraction - Collects specific information from website link hierarchies and extracts it into structured CSV or JSON formats.
- Domain-Restricted Crawling - Navigates website links and implements domain-restricted crawling to discover pages up to a specified depth.
- URL Traversal Queues - Manages the discovery of new pages and the overall traversal path using URL traversal queues.
- Distributed State Synchronization - Synchronizes crawl state and visited URL history across distributed server instances using a shared remote cache.
- Crawl Request Queues - Queues URLs for processing with priorities, delays, and retry logic using distributed crawl request queues.
- Priority-Based Request Queues - Organizes page traversal using priority-based request queues to optimize the order of website visits.
- JavaScript-Rendered Content Extractors - Extracts structured data from heavy pages by waiting for JavaScript-rendered content to load.
- Headless Crawlers - Uses a headless browser to navigate JavaScript-heavy applications and discover hidden endpoints.
- Distributed Crawl State Stores - Uses shared storage backends to track visited URLs and crawl progress across a cluster of workers.
- Structured Data Export Utilities - Streams collected information to external files using configurable field filters for CSV and JSON Lines.
- Stream-Based Exporters - Streams extracted page information directly into CSV or JSON Lines files for high-volume output.
- Client Environment Emulations - Simulates specific client environments by configuring user agents, viewport dimensions, and device profiles.
- Headless Browser Configurations - Configures rendering timeouts, viewport dimensions, and launch flags for headless browser instances.
- Session Concurrency Limiters - Implements controls to cap the number of simultaneous browser sessions to prevent system resource exhaustion.
- Distributed Crawl Workload Distribution - Partitions web scraping tasks across multiple server instances using a shared queue to increase throughput.
- Crawl Traversal Strategies - Organizes the sequence of page visits using depth-first search and breadth-first search algorithms.
- CDP Screenshot Captures - Captures page screenshots and metadata using the Chrome DevTools Protocol through custom evaluation functions.
- URL Request Tracking - Maintains a registry of fetched URLs to prevent redundant network requests and improve processing speed.
- Concurrent Request Limits - Restricts the number of simultaneous requests to manage system resources and prevent target server overload.
- Request Flow Control - Manages system stability through concurrency capping, delays, and request flow control.
- Robots Exclusion Compliance - Ensures adherence to site-specific crawling rules defined in robots.txt files.
- Robots.txt Compliance - Governs crawl paths by parsing site-specific access rules and sitemap.xml files.
- SEO and Compliance Scanning - Crawls websites while respecting robots.txt and sitemap.xml files to analyze site structure and indexing patterns.
- JavaScript Crawling Frameworks - Headless Chrome crawler with jQuery support.
- Web Scraping - Distributed crawler using headless Chrome.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
