Python3WebSpider/ProxyPool
ProxyPool
ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability.
The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a parsing method to extract host and port pairs. It also provides a lightweight HTTP API endpoint that returns a random high-scoring proxy, enabling client applications to fetch a verified proxy without direct database access. The collection, testing, and API server run as separate processes on configurable intervals, ensuring the pool remains continuously refreshed without manual intervention.
The project covers automated proxy collection from multiple sources, asynchronous health verification that updates scores based on response success or failure, and a sorted-set scoring model where proxies gain points on successful verification and lose points on failure. The API layer exposes a single endpoint for fetching a random working proxy, while the background scheduler manages the entire lifecycle of collection, testing, and serving.
Features
- Proxy Pool Automation - Automates the entire lifecycle of discovering, validating, and maintaining a pool of proxy servers via a web API.
- Proxy Availability Checks - Automates the verification of proxy availability against a target URL using an async HTTP client and updates scores based on response status.
- Pool Refresh Schedulers - Runs collector, tester, and API server as separate background processes on configurable intervals for continuous pool refresh.
- Reliability Scoring Systems - Assigns numeric scores to proxies, raising them on successful verification and lowering them on failure for priority selection.
- Proxy Health Verifications - Asynchronously tests proxy availability against a target URL and updates reliability scores based on response success or failure.
- Redis Sorted Set Priority Ordering - Stores proxies in a Redis sorted set with numeric scores that increase on success and decrease on failure for priority-based selection.
- Proxy Retrieval Endpoints - Exposes a single endpoint that returns a random high-scoring proxy for client consumption without database access.
- Single Proxy Fetches - Provides a dedicated endpoint to fetch a single validated proxy from the pool for immediate use.
- Proxy API Servers - Exposes a lightweight HTTP API endpoint that returns a random high-scoring proxy for client applications.
- Proxy Directory APIs - Exposes a lightweight HTTP endpoint that returns a random high-scoring proxy for client consumption without database access.
- Proxy List APIs - Provides a lightweight web endpoint that returns a single verified proxy from the pool without direct database access.
- Asynchronous Proxy Testers - Provides an asynchronous proxy tester that validates proxy availability against configurable target URLs.
- Proxy Source Aggregators - Combines diverse proxy links from multiple sources into a single stream using pluggable crawler methods.
- Pluggable Scraper Architectures - Implements a pluggable scraper architecture where new proxy sources are added via a common registration interface.
- Independent Process Launches - Launches scraper, verifier, and web server in independent background processes on separate schedules for automated pool maintenance.
- Proxy Crawler Frameworks - Ships a pluggable crawler framework that allows custom classes to extract proxy IP and port pairs from any target website.
- Proxy Verification Pipelines - Provides an async pipeline that verifies proxy availability and updates scores based on response success or failure.
- Crawler Extensions - Allows adding new proxy sources by writing a class that defines target URLs and a parse method to extract host and port.
