Awesome GitHub RepositoriesProxy Crawler Frameworks

This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web platforms, social media, and public repositories. It functions as a crawler framework that gathers proxy data and subscription links, a validation tool for testing server liveness, and a synchronization service for distributing the results. The system uses a plugin-based architecture that allows for the integration of custom Python scripts to handle diverse web source structures. It also includes utilities to transform raw proxy data into standardized configuration formats compatible w

ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability. The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a p

Haipproxy is a high-availability proxy gateway and distributed proxy pool manager. It consists of a system for storing and rotating verified IP proxy addresses using Redis, a web crawling system to discover anonymous proxies from public sources, and a validation engine that checks proxy functionality against specific target domains. The project implements a middleware layer that provides a stable entry point for requests by automatically rotating backend IP addresses. This includes a local proxy server that acts as a bridge between the client and the pool, decoupling the two by updating inter

IPProxyPool es un gestor de pool de proxies HTTP que rastrea, valida y sirve una lista rotativa de direcciones proxy funcionales a través de una API programática. Integra scraping de proxy, validación de conectividad y almacenamiento persistente en base de datos para proporcionar una fuente gestionada de direcciones IP para solicitudes de red. El sistema utiliza una arquitectura de scraping basada en plugins para recopilar direcciones IP de múltiples sitios web externos y una cola de validación asíncrona para probar estos candidatos en paralelo. Diferencia su pool asignando puntuaciones de estabilidad numéricas a los proxies a través de comprobaciones de salud periódicas y verificación de conectividad, permitiendo la recuperación de direcciones de alta estabilidad filtradas por región y protocolo. El proyecto incluye una capa de abstracción de base de datos con backends conectables para la persistencia de datos y proporciona herramientas para purgar entradas de proxy específicas. También admite lógica de validación personalizada para definir los criterios específicos utilizados para determinar la validez del proxy.