qiyeboy/IPProxyPool
IPProxyPool
IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy addresses via a programmatic API. It integrates proxy scraping, connectivity validation, and persistent database storage to provide a managed source of IP addresses for network requests.
The system uses a plugin-based scraping architecture to collect IP addresses from multiple external websites and an asynchronous validation queue to test these candidates in parallel. It differentiates its pool by assigning numeric stability scores to proxies through periodic health checks and connectivity verification, allowing the retrieval of high-stability addresses filtered by region and protocol.
The project includes a database abstraction layer with pluggable backends for data persistence and provides tools for purging specific proxy entries. It also supports custom validation logic to define the specific criteria used to determine proxy validity.
Features
- Proxy Pool Builders - Collects and manages a rotating pool of validated proxy IPs from public sources to ensure reliable connectivity.
- Proxy Availability Checks - Automates the verification of proxy reachability and status before admitting them into the pool.
- Proxy Address Storage - Implements specialized storage for persisting and indexing lists of validated network proxy servers.
- Proxy Retrieval Endpoints - Provides HTTP endpoints that return high-scoring proxies from a managed pool filtered by protocol and region.
- Proxy API Servers - Provides lightweight HTTP endpoints that return random high-scoring proxies from a managed pool for client use.
- HTTP Proxy Pool Managers - Manages the complete lifecycle of crawling, validating, and serving a rotating list of functional HTTP proxies via an API.
- Proxy Connectivity Testing - Implements tools for verifying the reachability and functional status of proxy servers to maintain high quality connections.
- Proxy List APIs - Provides programmatic interfaces for downloading and retrieving curated lists of validated proxy servers.
- Proxy Delivery Interfaces - Ships HTTP endpoints used to distribute validated proxy addresses from a managed pool to clients.
- Public API Exposure - Exposes internal system data and validated proxy lists as a secure backend for external applications.
- Proxy Stores - Provides a specialized storage layer for efficient organization and retrieval of validated proxy addresses and their status.
- Protocol Filters - Implements filtering mechanisms to organize and retrieve proxy addresses based on communication protocols like HTTP or SOCKS5.
- Reliability Scoring Systems - Assigns numeric stability scores to proxies based on health checks to prioritize the most reliable connections.
- Database Abstraction Layers - Provides a unified interface to interact with multiple database backends, decoupling core logic from storage.
- Plugin-Based - Uses a modular system of crawler plugins to extract proxy IP addresses from diverse external websites.
- Proxy Scraping Plugins - Ships a plugin-based system for crawling multiple proxy websites using defined extraction patterns.
- Asynchronous Task Queues - Implements a system for managing long-running proxy validation jobs through an asynchronous task queue.
- Proxy Crawler Frameworks - Implements a plugin-based framework to automate the gathering of proxy server data from the web.
- Health Checks - Runs recurring connectivity tests against live targets to monitor proxy availability and operational status.
- Scraping Infrastructure Management - Provides the infrastructure of managed proxies and rotating IPs necessary to bypass rate limits during large scale scraping.
