awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
qiyeboy avatar

qiyeboy/IPProxyPool

0
View on GitHub↗

IPProxyPool

IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy addresses via a programmatic API. It integrates proxy scraping, connectivity validation, and persistent database storage to provide a managed source of IP addresses for network requests.

The system uses a plugin-based scraping architecture to collect IP addresses from multiple external websites and an asynchronous validation queue to test these candidates in parallel. It differentiates its pool by assigning numeric stability scores to proxies through periodic health checks and connectivity verification, allowing the retrieval of high-stability addresses filtered by region and protocol.

The project includes a database abstraction layer with pluggable backends for data persistence and provides tools for purging specific proxy entries. It also supports custom validation logic to define the specific criteria used to determine proxy validity.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Proxy Pool Builders - Collects and manages a rotating pool of validated proxy IPs from public sources to ensure reliable connectivity.
  • Proxy Availability Checks - Automates the verification of proxy reachability and status before admitting them into the pool.
  • Proxy Address Storage - Implements specialized storage for persisting and indexing lists of validated network proxy servers.
  • Proxy Retrieval Endpoints - Provides HTTP endpoints that return high-scoring proxies from a managed pool filtered by protocol and region.
  • Proxy API Servers - Provides lightweight HTTP endpoints that return random high-scoring proxies from a managed pool for client use.
  • HTTP Proxy Pool Managers - Manages the complete lifecycle of crawling, validating, and serving a rotating list of functional HTTP proxies via an API.
  • Proxy Connectivity Testing - Implements tools for verifying the reachability and functional status of proxy servers to maintain high quality connections.
  • Proxy List APIs - Provides programmatic interfaces for downloading and retrieving curated lists of validated proxy servers.
  • Proxy Delivery Interfaces - Ships HTTP endpoints used to distribute validated proxy addresses from a managed pool to clients.
  • Public API Exposure - Exposes internal system data and validated proxy lists as a secure backend for external applications.
  • Proxy Stores - Provides a specialized storage layer for efficient organization and retrieval of validated proxy addresses and their status.
  • Protocol Filters - Implements filtering mechanisms to organize and retrieve proxy addresses based on communication protocols like HTTP or SOCKS5.
  • Reliability Scoring Systems - Assigns numeric stability scores to proxies based on health checks to prioritize the most reliable connections.
  • Database Abstraction Layers - Provides a unified interface to interact with multiple database backends, decoupling core logic from storage.
  • Plugin-Based - Uses a modular system of crawler plugins to extract proxy IP addresses from diverse external websites.
  • Proxy Scraping Plugins - Ships a plugin-based system for crawling multiple proxy websites using defined extraction patterns.
  • Asynchronous Task Queues - Implements a system for managing long-running proxy validation jobs through an asynchronous task queue.
  • Proxy Crawler Frameworks - Implements a plugin-based framework to automate the gathering of proxy server data from the web.
  • Health Checks - Runs recurring connectivity tests against live targets to monitor proxy availability and operational status.
  • Scraping Infrastructure Management - Provides the infrastructure of managed proxies and rotating IPs necessary to bypass rate limits during large scale scraping.
4,286 stars·1,277 forks·Python·43 views

Star history

Star history chart for qiyeboy/ipproxypoolStar history chart for qiyeboy/ipproxypool

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does qiyeboy/ipproxypool do?

IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy addresses via a programmatic API. It integrates proxy scraping, connectivity validation, and persistent database storage to provide a managed source of IP addresses for network requests.

What are the main features of qiyeboy/ipproxypool?

The main features of qiyeboy/ipproxypool are: Proxy Pool Builders, Proxy Availability Checks, Proxy Address Storage, Proxy Retrieval Endpoints, Proxy API Servers, HTTP Proxy Pool Managers, Proxy Connectivity Testing, Proxy List APIs.

Which projects share features with qiyeboy/ipproxypool?

Projects with overlapping indexed features include: jhao104/proxy_pool — This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through… python3webspider/proxypool — ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple… constverum/proxybroker — ProxyBroker is a tool for scraping public HTTP and SOCKS proxy addresses, validating their connectivity, and managing… mikechongcan/scylla — Scylla is a system for managing HTTP proxy pools and automating web extraction. It provides a specialized data… proxifly/free-proxy-list — This project is a public proxy aggregator and directory providing curated lists of validated HTTP and SOCKS proxy… wzdnzd/aggregator — This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web…

Projects sharing features with IPProxyPool

These projects share indexed features with IPProxyPool. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • jhao104/proxy_pooljhao104 avatar

    jhao104/proxy_pool

    23,776View on GitHub↗

    This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies. The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria

    Pythoncrawlerhttpproxy
    View on GitHub↗23,776
  • python3webspider/proxypoolPython3WebSpider avatar

    Python3WebSpider/ProxyPool

    6,223View on GitHub↗

    ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability. The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a p

    Pythonflaskhttpproxy
    View on GitHub↗6,223
  • constverum/proxybrokerconstverum avatar

    constverum/ProxyBroker

    4,150View on GitHub↗

    ProxyBroker is a tool for scraping public HTTP and SOCKS proxy addresses, validating their connectivity, and managing a curated pool of functional proxies. It consists of a proxy scraper for discovery, a validation engine to check anonymity and response times, and a pool manager to maintain a filtered queue of servers. The project includes a local rotating proxy server that acts as a single entry point, automatically distributing incoming network traffic across a pool of validated external proxies. This infrastructure allows for the rotation of IP addresses to maintain resilience during web d

    Pythonanonymityanonymouscrawler
    View on GitHub↗4,150
  • mikechongcan/scyllaMikeChongCan avatar

    MikeChongCan/scylla

    4,019View on GitHub↗

    Scylla is a system for managing HTTP proxy pools and automating web extraction. It provides a specialized data acquisition pipeline designed for gathering large-scale internet datasets for training and fine-tuning large language models. The project features a proxy rotation gateway that assigns fresh proxy addresses to incoming requests to mask origin traffic and avoid IP blocking. It includes a proxy pool manager that handles the collection, functional validation, and orchestration of proxy servers, complemented by a web dashboard for monitoring the health and geographic distribution of the

    Pythoncrawlerproxy-poolpython
    View on GitHub↗4,019
Compare all 30 related projects→