jhao104/proxy_pool
Proxy Pool
This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies.
The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria based on response codes and content length.
The manager handles background-scheduled collection and maintenance to ensure a consistent supply of functional addresses. It provides programmatic interfaces for retrieving, listing, and removing proxies, while using a key-value storage layer for high-speed retrieval and persistent state.
Network accessibility is managed through configurable listening IP addresses and ports.
Features
- Proxy Management Systems - Provides a comprehensive system to collect, validate, and serve free proxy IP addresses via an HTTP API.
- Proxy Pool Automation - Automates the collection and validation of free proxy addresses for consistent availability in a database.
- Proxy Availability Checks - Automates the gathering and verification of proxy server reachability through a background scheduler.
- Proxy Stores - Implements a persistent database-backed store using hashes for high-speed proxy IP retrieval.
- Proxy Request Routers - Exposes a REST interface that distributes outgoing requests across a pool of proxies to avoid IP bans.
- Proxy List APIs - Provides programmatic interfaces for retrieving curated lists of functional proxy servers.
- Proxy Scrapers - Provides an automated scraper to gather proxy addresses from multiple online sources for pool population.
- Proxy Delivery Interfaces - Implements an HTTP interface to serve valid proxy addresses from a managed pool to external clients.
- Key-Value Stores - Uses a Redis-based key-value store to maintain validated proxy addresses for high-speed retrieval.
- Proxy Address Storage - Saves validated proxy addresses into a database hash for persistent storage and high-speed retrieval.
- Proxy Connectivity Testing - Tests collected proxy addresses for connectivity to ensure only functional IPs are served.
- Proxy Pool Pruning - Removes unresponsive proxy addresses from the storage layer immediately upon receiving failure reports from clients.
- Proxy Source Aggregators - Combines diverse proxy links from custom discovery methods into a single managed stream.
- Dynamic Proxy Selection - Enables the selection of a single functional proxy IP from the pool for specific scraping tasks.
- Background Task Schedulers - Uses a timer-based background process to periodically scrape and refresh the pool of proxy addresses.
- Pluggable Scraper Architectures - Provides a pluggable architecture allowing new proxy discovery methods to be integrated via generator functions.
- Proxy Control APIs - Provides a programmable interface to manage and remove unresponsive proxies from the active pool.
- Recurring Maintenance Scheduling - Gathers free proxy addresses from multiple online sources on a schedule to maintain pool health.
