MikeChongCan/scylla
Scylla
Scylla is a system for managing HTTP proxy pools and automating web extraction. It provides a specialized data acquisition pipeline designed for gathering large-scale internet datasets for training and fine-tuning large language models.
The project features a proxy rotation gateway that assigns fresh proxy addresses to incoming requests to mask origin traffic and avoid IP blocking. It includes a proxy pool manager that handles the collection, functional validation, and orchestration of proxy servers, complemented by a web dashboard for monitoring the health and geographic distribution of the network.
The system supports dynamic content extraction using a headless browser crawler to capture data from websites that rely on JavaScript rendering. It also provides a JSON interface for proxy management and integration with external scraping frameworks.
Features
- Proxy Pool Builders - Manages the collection, functional validation, and orchestration of proxy IP addresses for large-scale data acquisition.
- LLM Dataset Collection - Provides a specialized pipeline for gathering large-scale internet datasets intended for LLM training and fine-tuning.
- Massive Corpora Acquisition - Implements a specialized infrastructure for high-volume retrieval of internet content for LLM datasets.
- HTTP Proxy Pool Managers - Automates the discovery, validation, and distribution of functional HTTP proxy servers.
- IP Address Masking - Masks the origin IP address of scraping bots by routing requests through a rotating set of proxies.
- Functional Verification - Performs automatic functional verification of proxy IP addresses to maintain a reliable scraping pool.
- Proxy Pool Automation - Automates the lifecycle of discovering and maintaining a pool of proxies for data extraction.
- Rotation Gateways - Ships a routing layer that automatically assigns a fresh proxy address to each incoming HTTP request.
- Proxy Traffic Management - Implements a traffic management mechanism that selects updated proxies for each request to mask origin traffic.
- JavaScript-Rendered Content Extractors - Uses headless browsers to extract structured data from websites that rely on JavaScript rendering.
- Headless Browsers - Utilizes headless browser engines to render and process JavaScript-driven dynamic web content.
- Headless Crawlers - Includes a headless crawler to capture dynamic content and bypass client-side rendering limitations.
- Scraping Infrastructure Management - Provides the infrastructure to manage rotated proxy pools and bypass blocking mechanisms during web extraction.
- Proxy List APIs - Provides a JSON API for retrieving filtered lists of proxy servers based on anonymity and location.
- Health Status Dashboards - Offers a visual dashboard for monitoring the real-time health and geographical spread of the proxy pool.
- Proxy Management Dashboards - Provides a web interface to monitor the health and geographic distribution of the proxy network.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
