lapwinglabs/x-ray
X Ray
X-Ray ist ein Web-Scraping-Framework und asynchroner Web-Crawler, der darauf ausgelegt ist, strukturierte Daten von Websites zu extrahieren. Es fungiert als HTML-Datenextraktor, der rohe Seiteninhalte mittels CSS-artiger Selektoren in ein definiertes Schema transformiert.
Das Projekt implementiert einen Headless-Browser-Crawler, der JavaScript ausführen kann, um dynamische Inhalte zu rendern. Es handhabt die Entdeckung von Website-Inhalten durch eine Breadth-First-Crawling-Strategie und automatische Paginierungserkennung, um mehrseitige Ergebnismengen zu durchlaufen.
Das Framework verwaltet Web-Daten-Pipelines mittels einer Concurrency-limitierten Request-Queue und Request-Rate-Control, um ausgehende Netzwerkanrufe zu regulieren. Extrahierte Ergebnisse werden über Stream-basierte Datenpersistenz verarbeitet, um große Datensätze ohne Überlastung des Systemspeichers zu bewältigen.
Features
- HTML Parsing and Extraction - Transforms raw HTML content into a predefined structured schema using CSS-style selectors.
- JavaScript Rendering - Renders dynamic content by executing JavaScript via headless browsers before the HTML parsing phase.
- Automated Web Scraping - Provides a framework for extracting structured information from websites at scale using automated crawlers.
- Multi-Page Crawling - Implements a breadth-first strategy to discover and collect data across multiple pages of a website.
- CSS Selector Data Extractors - Uses precise CSS selectors to transform raw HTML into a structured data format.
- Concurrent Request Limits - Manages the number of simultaneous outgoing requests to prevent overloading target servers.
- Request Rate Limiting - Controls the frequency and concurrency of outgoing HTTP requests to avoid server blocks.
- JavaScript-Rendered Content Extractors - Captures data from JavaScript-heavy websites by rendering pages via browser drivers before parsing.
- Headless Crawlers - Executes JavaScript using a headless browser to render dynamic content and discover links.
- Request-Browser Toggles - Provides the ability to switch between lightweight HTTP requests and full headless browser rendering for dynamic content.
- Web Crawlers - Implements a high-concurrency asynchronous crawler for visiting URLs and extracting structured data.
- Web Scraping Frameworks - Provides a programmable framework for automating data collection and managing browser-based navigation.
- Web Data Pipelines - Streams scraped information from multiple pages into files or databases to handle large datasets.
- Memory-Efficient Data Streaming - Writes extracted data to streams to process large datasets without overloading system memory.
- Data Stream Integrations - Exports extracted results via readable streams to ensure stability during long-running scraping tasks.
- Web Page Pagination Discovery - Implements automatic discovery of navigation links to traverse multi-page website result sets.
- Crawl Strategy Management - Supports configurable traversal orders, specifically implementing a breadth-first search strategy for content discovery.
- Pagination Crawlers - Automatically identifies and visits navigation links to traverse multi-page search results.
- Website Crawlers and Scrapers - Recursively retrieves content from across entire web domains by following links and pagination patterns.
- JavaScript Crawling Frameworks - Web scraper with pagination and crawler support.
- Web Scraping - Web scraping library with pagination support.
