constverum/ProxyBroker
ProxyBroker
ProxyBroker is a tool for scraping public HTTP and SOCKS proxy addresses, validating their connectivity, and managing a curated pool of functional proxies. It consists of a proxy scraper for discovery, a validation engine to check anonymity and response times, and a pool manager to maintain a filtered queue of servers.
The project includes a local rotating proxy server that acts as a single entry point, automatically distributing incoming network traffic across a pool of validated external proxies. This infrastructure allows for the rotation of IP addresses to maintain resilience during web data collection.
The system covers proxy quality assessment, including verification of support for cookies, referer headers, and post requests. It also provides attribute-based filtering by geography and protocol, as well as utilities for exporting discovered proxy lists to external files.
Features
- Proxy Rotation Services - Implements a local server that distributes network traffic across multiple proxy servers to maintain connectivity and avoid blocking.
- Local Middleware Proxies - Runs a local server that acts as a middleware entry point to rotate and forward traffic through a pool of external proxies.
- Proxy Availability Checks - Performs automated checks to verify the reachability and functional status of discovered proxies.
- Proxy Anonymity Validators - Checks the anonymity, response time, and connectivity of proxies against target URLs.
- Proxy Anonymity Assessment - Assesses proxies for anonymity levels, response times, and country of origin.
- Proxy Quality Metrics - Tests proxies for response times, anonymity levels, and support for specific headers to ensure high quality.
- HTTP Proxy Pool Managers - Automates the discovery, validation, and filtered management of functional HTTP proxy pools.
- Proxy Pool Builders - Maintains a rotating queue of validated proxy addresses to be distributed across network requests.
- Proxy Capability Verification - Verifies that proxies support critical capabilities like cookie handling and POST requests via target probes.
- Proxy Connectivity Testing - Tests discovered proxies against specific target servers to ensure they are functional and reachable.
- Proxy Feature Verification - Verifies proxy support for specific requirements such as cookies, referer headers, and POST requests.
- Proxy Scrapers - Automatically collects potential proxy addresses by scraping various public lists and web sources.
- Proxy List Aggregators - Automates the harvest, validation, and distribution of curated public proxy server lists.
- Proxy Attribute Filtering - Provides the ability to filter discovered proxies by protocol, country, and specific header support.
- Rule-Based Proxies - Restricts proxy selection based on a set of rules including protocol, anonymity level, and geography.
- Validation Pipelines - Implements a pipeline that concurrently validates multiple proxies against target servers to determine connectivity and anonymity.
- Scraping Infrastructure Management - Manages a reliable infrastructure of rotating IP addresses to prevent blocking during web data collection.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
