Proxy Pool

This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies.

The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria based on response codes and content length.

The manager handles background-scheduled collection and maintenance to ensure a consistent supply of functional addresses. It provides programmatic interfaces for retrieving, listing, and removing proxies, while using a key-value storage layer for high-speed retrieval and persistent state.

Network accessibility is managed through configurable listening IP addresses and ports.

Features

Proxy Management Systems - Provides a comprehensive system to collect, validate, and serve free proxy IP addresses via an HTTP API.

Proxy Pool Automation - Automates the collection and validation of free proxy addresses for consistent availability in a database.

Proxy Availability Checks - Automates the gathering and verification of proxy server reachability through a background scheduler.

Proxy Stores - Implements a persistent database-backed store using hashes for high-speed proxy IP retrieval.

Proxy Request Routers - Exposes a REST interface that distributes outgoing requests across a pool of proxies to avoid IP bans.

Proxy List APIs - Provides programmatic interfaces for retrieving curated lists of functional proxy servers.

Proxy Scrapers - Provides an automated scraper to gather proxy addresses from multiple online sources for pool population.

Proxy Delivery Interfaces - Implements an HTTP interface to serve valid proxy addresses from a managed pool to external clients.

Key-Value Stores - Uses a Redis-based key-value store to maintain validated proxy addresses for high-speed retrieval.

Proxy Address Storage - Saves validated proxy addresses into a database hash for persistent storage and high-speed retrieval.

Proxy Connectivity Testing - Tests collected proxy addresses for connectivity to ensure only functional IPs are served.

Proxy Pool Pruning - Removes unresponsive proxy addresses from the storage layer immediately upon receiving failure reports from clients.

Proxy Source Aggregators - Combines diverse proxy links from custom discovery methods into a single managed stream.

Dynamic Proxy Selection - Enables the selection of a single functional proxy IP from the pool for specific scraping tasks.

Background Task Schedulers - Uses a timer-based background process to periodically scrape and refresh the pool of proxy addresses.

Pluggable Scraper Architectures - Provides a pluggable architecture allowing new proxy discovery methods to be integrated via generator functions.

Proxy Control APIs - Provides a programmable interface to manage and remove unresponsive proxies from the active pool.

Recurring Maintenance Scheduling - Gathers free proxy addresses from multiple online sources on a schedule to maintain pool health.

jhao104proxy_pool

Features

Star history