lapwinglabs/x-ray
X Ray
X-Ray 是一个 Web 抓取框架和异步 Web 爬虫，旨在从网站中提取结构化数据。它作为一个 HTML 数据提取器，使用 CSS 样式选择器将原始页面内容转换为定义的模式。
该项目实现了一个能够执行 JavaScript 以渲染动态内容的无头浏览器爬虫。它通过广度优先爬取策略和自动分页发现来处理网站内容发现，以遍历多页结果集。
该框架使用并发限制的请求队列和请求速率控制来管理 Web 数据管线，以调节传出的网络调用。提取的结果通过基于流的数据持久化进行处理，以在不占用系统内存的情况下处理大数据集。
Features
- HTML Parsing and Extraction - Transforms raw HTML content into a predefined structured schema using CSS-style selectors.
- JavaScript Rendering - Renders dynamic content by executing JavaScript via headless browsers before the HTML parsing phase.
- Automated Web Scraping - Provides a framework for extracting structured information from websites at scale using automated crawlers.
- Multi-Page Crawling - Implements a breadth-first strategy to discover and collect data across multiple pages of a website.
- CSS Selector Data Extractors - Uses precise CSS selectors to transform raw HTML into a structured data format.
- Concurrent Request Limits - Manages the number of simultaneous outgoing requests to prevent overloading target servers.
- Request Rate Limiting - Controls the frequency and concurrency of outgoing HTTP requests to avoid server blocks.
- JavaScript-Rendered Content Extractors - Captures data from JavaScript-heavy websites by rendering pages via browser drivers before parsing.
- Headless Crawlers - Executes JavaScript using a headless browser to render dynamic content and discover links.
- Request-Browser Toggles - Provides the ability to switch between lightweight HTTP requests and full headless browser rendering for dynamic content.
- Web Crawlers - Implements a high-concurrency asynchronous crawler for visiting URLs and extracting structured data.
- Web Scraping Frameworks - Provides a programmable framework for automating data collection and managing browser-based navigation.
- Web Data Pipelines - Streams scraped information from multiple pages into files or databases to handle large datasets.
- Memory-Efficient Data Streaming - Writes extracted data to streams to process large datasets without overloading system memory.
- Data Stream Integrations - Exports extracted results via readable streams to ensure stability during long-running scraping tasks.
- Web Page Pagination Discovery - Implements automatic discovery of navigation links to traverse multi-page website result sets.
- Crawl Strategy Management - Supports configurable traversal orders, specifically implementing a breadth-first search strategy for content discovery.
- Pagination Crawlers - Automatically identifies and visits navigation links to traverse multi-page search results.
- Website Crawlers and Scrapers - Recursively retrieves content from across entire web domains by following links and pagination patterns.
- JavaScript Crawling Frameworks - Web scraper with pagination and crawler support.
- Web Scraping - Web scraping library with pagination support.
