awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
Python3WebSpider avatar

Python3WebSpider/ProxyPool

0
View on GitHub↗

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
proxypool.scrape.center
↗

ProxyPool

ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability.

The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a parsing method to extract host and port pairs. It also provides a lightweight HTTP API endpoint that returns a random high-scoring proxy, enabling client applications to fetch a verified proxy without direct database access. The collection, testing, and API server run as separate processes on configurable intervals, ensuring the pool remains continuously refreshed without manual intervention.

The project covers automated proxy collection from multiple sources, asynchronous health verification that updates scores based on response success or failure, and a sorted-set scoring model where proxies gain points on successful verification and lose points on failure. The API layer exposes a single endpoint for fetching a random working proxy, while the background scheduler manages the entire lifecycle of collection, testing, and serving.

Features

  • Proxy Pool Automation - Automates the entire lifecycle of discovering, validating, and maintaining a pool of proxy servers via a web API.
  • Proxy Availability Checks - Automates the verification of proxy availability against a target URL using an async HTTP client and updates scores based on response status.
  • Pool Refresh Schedulers - Runs collector, tester, and API server as separate background processes on configurable intervals for continuous pool refresh.
  • Reliability Scoring Systems - Assigns numeric scores to proxies, raising them on successful verification and lowering them on failure for priority selection.
6,223 stars·2,235 forks·Python·MIT·57 views
  • Proxy Health Verifications - Asynchronously tests proxy availability against a target URL and updates reliability scores based on response success or failure.
  • Redis Sorted Set Priority Ordering - Stores proxies in a Redis sorted set with numeric scores that increase on success and decrease on failure for priority-based selection.
  • Proxy Retrieval Endpoints - Exposes a single endpoint that returns a random high-scoring proxy for client consumption without database access.
  • Single Proxy Fetches - Provides a dedicated endpoint to fetch a single validated proxy from the pool for immediate use.
  • Proxy API Servers - Exposes a lightweight HTTP API endpoint that returns a random high-scoring proxy for client applications.
  • Proxy Directory APIs - Exposes a lightweight HTTP endpoint that returns a random high-scoring proxy for client consumption without database access.
  • Proxy List APIs - Provides a lightweight web endpoint that returns a single verified proxy from the pool without direct database access.
  • Asynchronous Proxy Testers - Provides an asynchronous proxy tester that validates proxy availability against configurable target URLs.
  • Proxy Source Aggregators - Combines diverse proxy links from multiple sources into a single stream using pluggable crawler methods.
  • Pluggable Scraper Architectures - Implements a pluggable scraper architecture where new proxy sources are added via a common registration interface.
  • Independent Process Launches - Launches scraper, verifier, and web server in independent background processes on separate schedules for automated pool maintenance.
  • Proxy Crawler Frameworks - Ships a pluggable crawler framework that allows custom classes to extract proxy IP and port pairs from any target website.
  • Proxy Verification Pipelines - Provides an async pipeline that verifies proxy availability and updates scores based on response success or failure.
  • Crawler Extensions - Allows adding new proxy sources by writing a class that defines target URLs and a parse method to extract host and port.

    • Star history

    Star history chart for python3webspider/proxypoolStar history chart for python3webspider/proxypool

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Projects sharing features with ProxyPool

    These projects share indexed features with ProxyPool. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • qiyeboy/ipproxypoolqiyeboy avatar

      qiyeboy/IPProxyPool

      4,286View on GitHub↗

      IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy addresses via a programmatic API. It integrates proxy scraping, connectivity validation, and persistent database storage to provide a managed source of IP addresses for network requests. The system uses a plugin-based scraping architecture to collect IP addresses from multiple external websites and an asynchronous validation queue to test these candidates in parallel. It differentiates its pool by assigning numeric stability scores to proxies through periodic health checks and conn

      Python
      View on GitHub↗4,286
    • jhao104/proxy_pooljhao104 avatar

      jhao104/proxy_pool

      23,776View on GitHub↗

      This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies. The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria

      Pythoncrawlerhttpproxy
      View on GitHub↗23,776
    • proxifly/free-proxy-listproxifly avatar

      proxifly/free-proxy-list

      3,865View on GitHub↗

      This project is a public proxy aggregator and directory providing curated lists of validated HTTP and SOCKS proxy servers. It features a machine-readable API service and tools designed for anonymous network routing and the automated rotation of outgoing IP addresses. The system distinguishes itself through a proxy rotation tool used to bypass rate limits and prevent detection by automated security systems. It provides a programmatic interface for retrieving and filtering verified proxies by country and protocol, delivering this data in JSON and text formats for integration into custom applica

      free-proxyfree-proxy-listfresh-proxies
      View on GitHub↗3,865
    • zu1k/proxypoolzu1k avatar

      zu1k/proxypool

      4,039View on GitHub↗

      Proxypool is an automated proxy crawler and aggregator that discovers, validates, and curates proxy servers from public pages and subscription addresses. It functions as a background service that collects proxy nodes across multiple protocols and serves the resulting validated list through a network API for external consumption. The system manages the full lifecycle of proxy discovery by aggregating data from multiple sources, deduplicating entries, and utilizing a connectivity validator to ensure only active and functional nodes are maintained. Crawl sources are managed via a configuration f

      Gocrawlerproxypool
      View on GitHub↗4,039
    Compare all 19 related projects→

    Frequently asked questions

    What does python3webspider/proxypool do?

    ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability.

    What are the main features of python3webspider/proxypool?

    The main features of python3webspider/proxypool are: Proxy Pool Automation, Proxy Availability Checks, Pool Refresh Schedulers, Reliability Scoring Systems, Proxy Health Verifications, Redis Sorted Set Priority Ordering, Proxy Retrieval Endpoints, Single Proxy Fetches.

    Which projects share features with python3webspider/proxypool?

    Projects with overlapping indexed features include: qiyeboy/ipproxypool — IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy… jhao104/proxy_pool — This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through… proxifly/free-proxy-list — This project is a public proxy aggregator and directory providing curated lists of validated HTTP and SOCKS proxy… zu1k/proxypool — Proxypool is an automated proxy crawler and aggregator that discovers, validates, and curates proxy servers from… thespeedx/proxy-list — PROXY-List is a public proxy aggregator that provides data structures for storing and aggregating publicly available… mikechongcan/scylla — Scylla is a system for managing HTTP proxy pools and automating web extraction. It provides a specialized data…