Open-source alternatives to PROXY List

This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies. The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria

Oblivion is an encrypted tunneling service and internet privacy tool designed to route network traffic through private encrypted tunnels. It functions as a network privacy gateway that masks user identity and encrypts web traffic to prevent tracking and unauthorized surveillance. The project acts as a secure network proxy that combines end-to-end encryption with latency-optimized routing to increase data throughput and reduce network lag. This infrastructure allows for the masking of a device's public identity by relaying connections through a private network. The system provides capabilitie

This is a collection of Python scripts designed for extracting data from popular Chinese websites and mobile applications. It functions as a multi-platform data extraction toolkit, capable of automating tasks such as downloading videos from platforms like Bilibili and Douyin, scraping product reviews and images from e-commerce sites like Taobao and JD.com, and booking train tickets on the 12306 railway system. The project distinguishes itself through its focus on automating specific, high-value tasks within the Chinese internet ecosystem. It includes capabilities for solving Chinese CAPTCHA c

This project is a self-hosted email verification system and API designed to validate email existence and clean mailing lists on private infrastructure. It functions as a deliverability tool that confirms if email addresses are reachable by communicating with mail servers via the SMTP protocol without sending actual messages. The system is distinguished by its high-volume SMTP infrastructure, which utilizes a stateless worker architecture and message queue task distribution to scale validation tasks. It includes an SMTP proxy gateway that routes requests through SOCKS5 proxies to mask server i

Proxypool is an automated proxy crawler and aggregator that discovers, validates, and curates proxy servers from public pages and subscription addresses. It functions as a background service that collects proxy nodes across multiple protocols and serves the resulting validated list through a network API for external consumption. The system manages the full lifecycle of proxy discovery by aggregating data from multiple sources, deduplicating entries, and utilizing a connectivity validator to ensure only active and functional nodes are maintained. Crawl sources are managed via a configuration f

IPProxyPool is an HTTP proxy pool manager that crawls, validates, and serves a rotating list of functional proxy addresses via a programmatic API. It integrates proxy scraping, connectivity validation, and persistent database storage to provide a managed source of IP addresses for network requests. The system uses a plugin-based scraping architecture to collect IP addresses from multiple external websites and an asynchronous validation queue to test these candidates in parallel. It differentiates its pool by assigning numeric stability scores to proxies through periodic health checks and conn

This project is a Sina Weibo web scraper and social media data pipeline designed to extract user profiles, posts, comments, and multimedia assets. It functions as a containerized data crawler that automates the collection and local storage of social media content and engagement metrics. The system includes a processing layer that uses large language models to analyze scraped text, generating summaries and sentiment analysis. It differentiates itself through a deployment-ready container model featuring an HTTP interface for managing extraction tasks and monitoring job progress. The crawler co

This project is a repository of proxy server addresses, providing a regularly updated dataset of HTTP, SOCKS4, and SOCKS5 endpoints. It functions as a directory for applications that require managed network routing, offering curated connection details to facilitate traffic management and infrastructure tasks. The system distinguishes itself through an automated harvesting pipeline that refreshes the proxy list on an hourly schedule. This process includes geospatial metadata enrichment, which appends location data to each entry, allowing for regional routing and the simulation of local user ac

This project is a collection of Python scripts and tools designed for web scraping, browser automation, and large-scale data extraction. It provides a set of implementations for retrieving information from websites and private APIs, including tools for multimedia downloading and social media data archiving. The toolset includes specialized mechanisms for bypassing anti-scraping measures through IP proxy pool rotation and multi-threaded crawlers. It also features capabilities for simulating browser sessions to handle authentication, intercepting session cookies, and decrypting network payloads

Jsoup is a Java library designed for parsing, extracting, and manipulating HTML and XML content. It provides a document object model that represents web content as a hierarchical tree, allowing for programmatic navigation and modification of elements, attributes, and text. The library functions as a toolkit for web scraping, enabling the retrieval of remote content via standard web protocols and the management of HTTP sessions for automated form interaction. The library distinguishes itself through its fault-tolerant tokenization, which reconstructs valid document structures from malformed or

Newspaper is a Python library designed for scraping, parsing, and analyzing web-based information. It functions as a framework for automated news aggregation and large-scale web content extraction, providing tools to download, clean, and structure text, metadata, and media from diverse online sources. The project distinguishes itself through a pipeline-oriented architecture that combines heuristic-based content extraction with natural language processing. It automatically identifies and isolates article bodies from web page boilerplate while simultaneously performing language detection, keywo

NoMoreWalls is a proxy subscription aggregator and configuration generator designed to bypass internet censorship. It functions as a tool for collecting public proxy nodes and compiling them into subscription files compatible with the Clash network core. The project differentiates itself by providing a local system for fetching, filtering, and provisioning curated proxy server configurations from multiple sources. It includes mechanisms to exclude specific proxy protocols and limit imported nodes to improve network stability and performance. The system manages network traffic routing through

MechanicalSoup is a Python web automation library designed to simulate browser behavior. It functions as a toolkit for web scraping and automation, providing an HTML parsing engine and an HTTP session manager to interact with websites programmatically. The library enables headless web interaction by mimicking a real user session. It manages persistent state through cookie handling and automatic redirect following, allowing for programmatic website navigation and the simulation of complex browser interactions. Its capabilities cover automated form population and submission using CSS selectors

Obscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for AI models to control browser sessions, interact with websites, and extract web data using a WebSocket implementation of the Chrome DevTools Protocol. The project focuses on bot detection evasion by randomizing browser fingerprints, masking native functions, and blocking tracking scripts to mimic human behavior. It further secures identities through a traffic layer that routes network requests via HTTP or SOCKS5 proxies. The system supports large-scale data extraction through

This is a serverless proxy subscription converter implemented as a Cloudflare Worker. It aggregates multiple proxy subscriptions into a single unified stream and converts them into client-specific configuration formats through protocol translation. The project functions as a proxy protocol translator and routing rule generator, enabling the definition and export of custom network traffic rules and policy groups. It also includes a URL shortener that utilizes key-value storage to create shortened aliases for long subscription links. The system covers a broad range of subscription management c

DotnetSpider is a .NET web crawling framework and C# data extraction tool designed for automated web page discovery and the retrieval of structured data from the internet at scale. It functions as a high-level web scraping library for collecting information from various websites. The framework provides capabilities for automated web crawling and large-scale data scraping. It enables web content extraction to facilitate the creation of local databases or the analysis of online information through programmatic web automation within the .NET ecosystem. The system utilizes a pipeline-based data

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

AutoMergePublicNodes is a system for scraping public proxy nodes from across the internet and merging them into a unified list. It serves as an aggregator and converter that transforms these collected nodes into specialized subscription formats for various network routing clients. The project includes dedicated generators for creating configuration files and subscriptions compatible with Clash, Clash Meta, Sing-Box, and V2Ray. It handles the conversion of aggregated lists into Base64 and other standardized formats required by these specific network applications. The software also provides a

youtube-transcript-api is a Python library designed to retrieve and download subtitles and captions from YouTube videos using video IDs. It functions as an API client that extracts text and timing data for video content. The project includes a wrapper for automated translation, allowing transcripts to be converted into different target languages. It also features a retrieval system that supports routing requests through HTTP, HTTPS, or SOCKS proxies to avoid IP blocking and regional restrictions. The library provides tools for identifying available subtitle tracks and converting raw transcri

wool_scripts is a collection of HTTP proxy rule sets, network traffic filters, and automation scripts designed for mobile applications. It functions as an application proxy router to reroute specific traffic and AI model requests through proxy servers to change IP addresses. The project provides a system for intercepting network requests to remove embedded advertisements from mobile applications and mini-programs. It also includes filters to suppress operating system update notifications and automatic software downloads by blocking requests to update servers. Beyond traffic filtering, the re

AnyCrawl is an AI-powered data extractor, automated web crawler, and headless browser orchestrator. It serves as a web content extraction API and a gateway that connects crawling and scraping tools to language models using a standardized API protocol. The project specializes in converting unstructured website content into structured JSON or markdown optimized for AI assistants. It utilizes language models and JSON schemas to pull specific information into validated formats and provides capabilities for AI page summarization and LLM-optimized content extraction. The system manages comprehensi

Webmagic is a Java web crawling framework designed for building scalable automated crawlers to download and process large volumes of web pages. It functions as a distributed web crawler and dynamic content crawler, utilizing an XPath HTML parser to locate and extract specific data points from page structures. The framework distinguishes itself through its ability to handle dynamic content by rendering JavaScript and executing asynchronous requests to extract data from non-static pages. It also allows users to define and execute crawler logic via scripting languages, enabling the update of col

Gost is a Go-based network tunnel and multi-protocol proxy server. It functions as a gateway for routing TCP and UDP traffic, creating secure network tunnels between remote endpoints, and acting as a DNS proxy server to resolve domain name queries. The project is distinguished by its ability to implement multi-hop proxy chaining, which links multiple network nodes in a sequence to route traffic through specific paths. It also provides transparent proxying by integrating with virtual network interfaces to intercept system-level traffic without requiring manual client configuration. The system

This project is a Model Context Protocol server that enables Large Language Models to control Playwright browsers for web automation, scraping, and end-to-end testing. It functions as a programmable interface for executing JavaScript, capturing screenshots, and interacting with web elements across multiple browser engines. The server exposes browser automation capabilities as a set of standardized tools that models can discover and invoke. It supports session-based browser isolation to ensure unique contexts for each client connection and provides a transport layer using either standard input

Pup is a command line tool for extracting and filtering data from HTML documents using CSS selectors. It functions as a parser and selector engine that isolates specific elements based on tags, IDs, classes, and attributes. The project provides utilities for converting selected HTML nodes into plain text, attribute values, or structured JSON objects. It includes a markup formatter that corrects missing tags and applies consistent indentation to improve the readability of HTML documents. The tool handles the retrieval of text content and attributes through a CSS selector engine, supporting co

This is a collection of Python automation scripts and utility tools designed to handle repetitive technical tasks, system administration, and developer workflows. The project serves as a suite for task automation, data utility, and web automation. The collection includes specialized tools for multimedia processing, such as optical character recognition for extracting text from images, speech-to-text conversion, and real-time face and human body detection. It also features web scraping and monitoring capabilities to track product prices, fetch external API content, and automate interactions wi

req is a chainable HTTP client library for Go designed to simplify request configuration and automatic response decoding into structures. It provides a fluent-interface request builder that allows developers to incrementally define request properties and encapsulate HTTP logic into reusable API SDKs. The project distinguishes itself with a TLS fingerprint emulator that mimics browser network signatures to bypass bot detection and crawler filters. It also includes a concurrent file downloader that increases transfer speeds by fetching large remote files in parallel segments. The library cover