qiyeboy/IPProxyPool
IPProxyPool
IPProxyPool est un gestionnaire de pool de proxy HTTP qui crawle, valide et sert une liste rotative d'adresses proxy fonctionnelles via une API programmatique. Il intègre le scraping de proxy, la validation de connectivité et le stockage en base de données persistante pour fournir une source gérée d'adresses IP pour les requêtes réseau.
Le système utilise une architecture de scraping basée sur des plugins pour collecter des adresses IP depuis plusieurs sites web externes et une file d'attente de validation asynchrone pour tester ces candidats en parallèle. Il différencie son pool en attribuant des scores de stabilité numériques aux proxys via des vérifications de santé périodiques et une vérification de connectivité, permettant la récupération d'adresses à haute stabilité filtrées par région et protocole.
Le projet inclut une couche d'abstraction de base de données avec des backends enfichables pour la persistance des données et fournit des outils pour purger des entrées proxy spécifiques. Il prend également en charge une logique de validation personnalisée pour définir les critères spécifiques utilisés pour déterminer la validité d'un proxy.
Features
- Proxy Pool Builders - Collects and manages a rotating pool of validated proxy IPs from public sources to ensure reliable connectivity.
- Proxy Availability Checks - Automates the verification of proxy reachability and status before admitting them into the pool.
- Proxy Address Storage - Implements specialized storage for persisting and indexing lists of validated network proxy servers.
- Proxy Retrieval Endpoints - Provides HTTP endpoints that return high-scoring proxies from a managed pool filtered by protocol and region.
- Proxy API Servers - Provides lightweight HTTP endpoints that return random high-scoring proxies from a managed pool for client use.
- HTTP Proxy Pool Managers - Manages the complete lifecycle of crawling, validating, and serving a rotating list of functional HTTP proxies via an API.
- Proxy Connectivity Testing - Implements tools for verifying the reachability and functional status of proxy servers to maintain high quality connections.
- Proxy List APIs - Provides programmatic interfaces for downloading and retrieving curated lists of validated proxy servers.
- Proxy Delivery Interfaces - Ships HTTP endpoints used to distribute validated proxy addresses from a managed pool to clients.
- Public API Exposure - Exposes internal system data and validated proxy lists as a secure backend for external applications.
- Proxy Stores - Provides a specialized storage layer for efficient organization and retrieval of validated proxy addresses and their status.
- Protocol Filters - Implements filtering mechanisms to organize and retrieve proxy addresses based on communication protocols like HTTP or SOCKS5.
- Reliability Scoring Systems - Assigns numeric stability scores to proxies based on health checks to prioritize the most reliable connections.
- Database Abstraction Layers - Provides a unified interface to interact with multiple database backends, decoupling core logic from storage.
- Plugin-Based - Uses a modular system of crawler plugins to extract proxy IP addresses from diverse external websites.
- Proxy Scraping Plugins - Ships a plugin-based system for crawling multiple proxy websites using defined extraction patterns.
- Asynchronous Task Queues - Implements a system for managing long-running proxy validation jobs through an asynchronous task queue.
- Proxy Crawler Frameworks - Implements a plugin-based framework to automate the gathering of proxy server data from the web.
- Health Checks - Runs recurring connectivity tests against live targets to monitor proxy availability and operational status.
- Scraping Infrastructure Management - Provides the infrastructure of managed proxies and rotating IPs necessary to bypass rate limits during large scale scraping.
