# zu1k/proxypool

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/zu1k-proxypool).**

4,017 stars · 2,390 forks · Go · GPL-3.0

## Links

- GitHub: https://github.com/zu1k/proxypool
- awesome-repositories: https://awesome-repositories.com/repository/zu1k-proxypool.md

## Topics

`crawler` `proxypool`

## Description

Proxypool is an automated proxy crawler and aggregator that discovers, validates, and curates proxy servers from public pages and subscription addresses. It functions as a background service that collects proxy nodes across multiple protocols and serves the resulting validated list through a network API for external consumption.

The system manages the full lifecycle of proxy discovery by aggregating data from multiple sources, deduplicating entries, and utilizing a connectivity validator to ensure only active and functional nodes are maintained. Crawl sources are managed via a configuration file to target specific external addresses.

The project handles continuous proxy list management through scheduled background tasks that automate the refreshing and updating of available nodes. This process includes automated connectivity testing and the pruning of inactive servers to keep the curated list current.

## Tags

### Networking & Communication

- [Proxy Node Aggregators](https://awesome-repositories.com/f/networking-communication/proxy-node-aggregators.md) — Aggregates curated collections of proxy server addresses and subscription links across multiple protocols. ([source](https://github.com/zu1k/proxypool#readme))
- [Proxy Pool Automation](https://awesome-repositories.com/f/networking-communication/proxy-pool-automation.md) — Provides a comprehensive system for the automated discovery, validation, and maintenance of a proxy server pool.
- [Proxy Connectivity Testing](https://awesome-repositories.com/f/networking-communication/proxy-connectivity-testing.md) — Implements automated testing to verify the reachability and functional status of discovered proxy servers.
- [Public Proxy Directories](https://awesome-repositories.com/f/networking-communication/public-proxy-directories.md) — Builds a curated directory of validated public proxy server addresses by crawling multiple sources. ([source](https://github.com/zu1k/proxypool/blob/master/main.go))
- [Public Proxy Discovery](https://awesome-repositories.com/f/networking-communication/public-proxy-discovery.md) — Finds usable proxy servers across multiple protocols by automatically scanning public pages and subscription addresses.
- [Proxy List Aggregators](https://awesome-repositories.com/f/networking-communication/socks-proxies/proxy-list-aggregators.md) — Automates the harvesting, validation, and distribution of curated public proxy server lists.
- [Network Exposure Tools](https://awesome-repositories.com/f/networking-communication/network-exposure-tools.md) — Serves the validated list of proxy nodes through a network endpoint for external consumption.
- [Proxy List APIs](https://awesome-repositories.com/f/networking-communication/proxy-list-apis.md) — Provides a programmatic interface for downloading and retrieving curated lists of validated proxy servers. ([source](https://github.com/zu1k/proxypool/blob/master/main.go))

### Artificial Intelligence & ML

- [Connectivity Validations](https://awesome-repositories.com/f/artificial-intelligence-ml/external-server-connectivity/server-connection-managers/custom-server-connections/local-ai-endpoint-connections/connectivity-validations.md) — Performs connectivity validations to ensure proxy nodes are active and functional before inclusion. ([source](https://github.com/zu1k/proxypool/blob/master/README.md))

### Data & Databases

- [Multi-Source Content Aggregation](https://awesome-repositories.com/f/data-databases/multi-source-content-aggregation.md) — Collects and merges proxy nodes from multiple public pages and channels into a single curated list.

### Development Tools & Productivity

- [Background Task Schedulers](https://awesome-repositories.com/f/development-tools-productivity/background-task-schedulers.md) — Runs automated background tasks on a fixed time interval to discover new proxy nodes.
- [Pool Refresh Schedulers](https://awesome-repositories.com/f/development-tools-productivity/background-task-schedulers/continuous-process-executors/pool-refresh-schedulers.md) — Continuously refreshes the proxy pool by running collector and tester processes as background tasks. ([source](https://github.com/zu1k/proxypool/blob/master/main.go))
- [Daily Proxy Node Refresh Schedulers](https://awesome-repositories.com/f/development-tools-productivity/cron-scheduling/feed-refresh-scheduling/daily-proxy-node-refresh-schedulers.md) — Automatically refreshes proxy node lists on a fixed schedule to maintain current working servers. ([source](https://github.com/zu1k/proxypool/blob/master/README.md))

### DevOps & Infrastructure

- [Node Availability Verification](https://awesome-repositories.com/f/devops-infrastructure/node-availability-verification.md) — Ensures discovered proxy nodes are functional and available for use by testing them automatically. ([source](https://github.com/zu1k/proxypool#readme))

### Part of an Awesome List

- [Proxy Pools](https://awesome-repositories.com/f/awesome-lists/data/proxy-pools.md) — A centralized pool for managing and distributing proxy nodes.