MikeChongCan/scylla
Scylla
Scylla ist ein System zur Verwaltung von HTTP-Proxy-Pools und zur Automatisierung der Web-Extraktion. Es bietet eine spezialisierte Datenerfassungspipeline, die für das Sammeln großskaliger Internet-Datensätze zum Training und zur Feinabstimmung von Large Language Models entwickelt wurde.
Das Projekt verfügt über ein Proxy-Rotation-Gateway, das eingehenden Anfragen frische Proxy-Adressen zuweist, um den Ursprungs-Traffic zu maskieren und IP-Blockaden zu vermeiden. Es enthält einen Proxy-Pool-Manager, der die Sammlung, funktionale Validierung und Orchestrierung von Proxyservern handhabt, ergänzt durch ein Web-Dashboard zur Überwachung der Gesundheit und geografischen Verteilung des Netzwerks.
Das System unterstützt dynamische Content-Extraktion mittels Headless-Browser-Crawler, um Daten von Websites zu erfassen, die auf JavaScript-Rendering basieren. Es bietet zudem eine JSON-Schnittstelle für die Proxy-Verwaltung und Integration in externe Scraping-Frameworks.
Features
- Proxy Pool Builders - Manages the collection, functional validation, and orchestration of proxy IP addresses for large-scale data acquisition.
- LLM Dataset Collection - Provides a specialized pipeline for gathering large-scale internet datasets intended for LLM training and fine-tuning.
- Massive Corpora Acquisition - Implements a specialized infrastructure for high-volume retrieval of internet content for LLM datasets.
- HTTP Proxy Pool Managers - Automates the discovery, validation, and distribution of functional HTTP proxy servers.
- IP Address Masking - Masks the origin IP address of scraping bots by routing requests through a rotating set of proxies.
- Functional Verification - Performs automatic functional verification of proxy IP addresses to maintain a reliable scraping pool.
- Proxy Pool Automation - Automates the lifecycle of discovering and maintaining a pool of proxies for data extraction.
- Rotation Gateways - Ships a routing layer that automatically assigns a fresh proxy address to each incoming HTTP request.
- Proxy Traffic Management - Implements a traffic management mechanism that selects updated proxies for each request to mask origin traffic.
- JavaScript-Rendered Content Extractors - Uses headless browsers to extract structured data from websites that rely on JavaScript rendering.
- Headless Browsers - Utilizes headless browser engines to render and process JavaScript-driven dynamic web content.
- Headless Crawlers - Includes a headless crawler to capture dynamic content and bypass client-side rendering limitations.
- Scraping Infrastructure Management - Provides the infrastructure to manage rotated proxy pools and bypass blocking mechanisms during web extraction.
- Proxy List APIs - Provides a JSON API for retrieving filtered lists of proxy servers based on anonymity and location.
- Health Status Dashboards - Offers a visual dashboard for monitoring the real-time health and geographical spread of the proxy pool.
- Proxy Management Dashboards - Provides a web interface to monitor the health and geographic distribution of the proxy network.
