matthewmueller/x-ray
X Ray
X-ray is a headless browser web scraper and HTML content crawler designed to extract structured data from websites. It functions as a stream-based data scraper and structured data extractor, using selectors to retrieve text and attributes from HTML as nested objects or arrays.
The project includes a request rate controller to manage network traffic through concurrency limits, throttles, and timeouts. It handles dynamic website scraping by rendering JavaScript via a headless browser and performs automated website crawling using breadth-first link following and pagination management.
The system provides a data pipeline that applies functional value transformations to raw strings and writes results to a readable stream to prevent memory overflow during large-scale web scraping jobs.
Features
- Headless Browsers - Uses a headless browser to render JavaScript and extract content from dynamic web pages.
- Dynamic - Employs techniques to retrieve data from websites that rely on JavaScript for content rendering.
- HTML Selector Extractors - Provides a selector-based parser to retrieve text and attributes from HTML as structured nested objects or arrays.
- Multi-Page Crawling - Implements systematic site discovery by navigating through sequential pages and link hierarchies.
- Structured Data Extraction - Extracts specific data points from HTML into structured formats like objects and arrays.
- Web Data Extraction - Programmatically scrapes and processes web content to create structured data objects.
- Data Extractions - Retrieves specific text and attributes from HTML using CSS selectors and DOM traversal.
- Recursive Crawlers - Discovers and indexes linked pages from seed URLs using recursive breadth-first traversal.
- Extraction Streams - Streams extracted structured data incrementally to prevent data loss during long-running scraping jobs.
- Data Transformation Functions - Applies transformation functions to scrape results for trimming, slicing, and data cleaning.
- Text Transformation Functions - Provides custom filter functions to clean and modify raw scraped strings before final output.
- Web Page Pagination Discovery - Automatically identifies and follows pagination links to traverse multi-page HTML result sets.
- Scraping Result Streams - Writes extracted results to a readable stream or file immediately to prevent memory overflow.
- Stream-Based Data Pipelines - Uses a processing architecture that pipes extracted data directly to storage to optimize memory usage.
- Concurrent Request Limits - Controls the number of simultaneous outgoing requests to prevent overloading target servers.
- Request Rate Limiting - Manages the frequency and concurrency of outgoing requests to avoid server detection or overload.
- Large-Scale Domain Crawlers - Provides infrastructure for comprehensive discovery and indexing of entire websites at scale.
- Parsing Utilities - Web scraping utility.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
