constverum/ProxyBroker
ProxyBroker
ProxyBroker este un instrument pentru scraping-ul adreselor proxy publice HTTP și SOCKS, validarea conectivității acestora și gestionarea unui pool curatoriat de proxy-uri funcționale. Acesta constă într-un scraper de proxy-uri pentru descoperire, un motor de validare pentru a verifica anonimatul și timpii de răspuns, și un manager de pool pentru a menține o coadă filtrată de servere.
Proiectul include un server proxy rotativ local care acționează ca un punct unic de intrare, distribuind automat traficul de rețea primit către un pool de proxy-uri externe validate. Această infrastructură permite rotația adreselor IP pentru a menține reziliența în timpul colectării datelor web.
Sistemul acoperă evaluarea calității proxy-urilor, inclusiv verificarea suportului pentru cookie-uri, headere referer și cereri post. Oferă, de asemenea, filtrare bazată pe atribute după geografie și protocol, precum și utilitare pentru exportul listelor de proxy-uri descoperite în fișiere externe.
Features
- Proxy Rotation Services - Implements a local server that distributes network traffic across multiple proxy servers to maintain connectivity and avoid blocking.
- Local Middleware Proxies - Runs a local server that acts as a middleware entry point to rotate and forward traffic through a pool of external proxies.
- Proxy Availability Checks - Performs automated checks to verify the reachability and functional status of discovered proxies.
- Proxy Anonymity Validators - Checks the anonymity, response time, and connectivity of proxies against target URLs.
- Proxy Anonymity Assessment - Assesses proxies for anonymity levels, response times, and country of origin.
- Proxy Quality Metrics - Tests proxies for response times, anonymity levels, and support for specific headers to ensure high quality.
- HTTP Proxy Pool Managers - Automates the discovery, validation, and filtered management of functional HTTP proxy pools.
- Proxy Pool Builders - Maintains a rotating queue of validated proxy addresses to be distributed across network requests.
- Proxy Capability Verification - Verifies that proxies support critical capabilities like cookie handling and POST requests via target probes.
- Proxy Connectivity Testing - Tests discovered proxies against specific target servers to ensure they are functional and reachable.
- Proxy Feature Verification - Verifies proxy support for specific requirements such as cookies, referer headers, and POST requests.
- Proxy Scrapers - Automatically collects potential proxy addresses by scraping various public lists and web sources.
- Proxy List Aggregators - Automates the harvest, validation, and distribution of curated public proxy server lists.
- Proxy Attribute Filtering - Provides the ability to filter discovered proxies by protocol, country, and specific header support.
- Rule-Based Proxies - Restricts proxy selection based on a set of rules including protocol, anonymity level, and geography.
- Validation Pipelines - Implements a pipeline that concurrently validates multiple proxies against target servers to determine connectivity and anonymity.
- Scraping Infrastructure Management - Manages a reliable infrastructure of rotating IP addresses to prevent blocking during web data collection.
