Projects sharing features with ProxyBroker

This project is a Python-based proxy pool manager that collects, validates, and serves free proxy IP addresses through an HTTP API. It consists of an automated scraper to gather addresses from multiple online sources, a persistent database-backed store for organization, and a delivery interface for retrieving validated proxies. The system features a pluggable scraper architecture that allows for the integration of custom discovery methods and source expansion via generator functions. It employs decorator-based validation logic, enabling the definition of custom connectivity and HTTPS criteria

This project is a public proxy aggregator and directory providing curated lists of validated HTTP and SOCKS proxy servers. It features a machine-readable API service and tools designed for anonymous network routing and the automated rotation of outgoing IP addresses. The system distinguishes itself through a proxy rotation tool used to bypass rate limits and prevent detection by automated security systems. It provides a programmatic interface for retrieving and filtering verified proxies by country and protocol, delivering this data in JSON and text formats for integration into custom applica

Proxypool is an automated proxy crawler and aggregator that discovers, validates, and curates proxy servers from public pages and subscription addresses. It functions as a background service that collects proxy nodes across multiple protocols and serves the resulting validated list through a network API for external consumption. The system manages the full lifecycle of proxy discovery by aggregating data from multiple sources, deduplicating entries, and utilizing a connectivity validator to ensure only active and functional nodes are maintained. Crawl sources are managed via a configuration f

AnyCrawl is an AI-powered data extractor, automated web crawler, and headless browser orchestrator. It serves as a web content extraction API and a gateway that connects crawling and scraping tools to language models using a standardized API protocol. The project specializes in converting unstructured website content into structured JSON or markdown optimized for AI assistants. It utilizes language models and JSON schemas to pull specific information into validated formats and provides capabilities for AI page summarization and LLM-optimized content extraction. The system manages comprehensi

cloudscraper is a Python library designed to bypass Cloudflare anti-bot protections by resolving JavaScript challenges and mimicking browser fingerprints. It functions as a specialized tool for accessing websites that employ automated security systems to block scripts and headless browsers. The project differentiates itself through the use of interchangeable JavaScript runtimes, such as Node.js or V8, to execute challenge code and obtain security clearance tokens. It employs a fingerprint rotation engine and HTTP request emulation to rotate browser headers and device identifiers, mimicking hu

Haipproxy is a high-availability proxy gateway and distributed proxy pool manager. It consists of a system for storing and rotating verified IP proxy addresses using Redis, a web crawling system to discover anonymous proxies from public sources, and a validation engine that checks proxy functionality against specific target domains. The project implements a middleware layer that provides a stable entry point for requests by automatically rotating backend IP addresses. This includes a local proxy server that acts as a bridge between the client and the pool, decoupling the two by updating inter

Botasaurus is a Python web scraping framework and headless browser automation system used to build scalable data extraction tools. It functions as a web data extraction tool and OCR document parser, converting website content, images, and PDF files into structured formats such as JSON, CSV, and Excel. The framework distinguishes itself by providing a scraper management interface that allows Python functions to be wrapped in a web-based UI or deployed as standalone desktop applications. This enables non-technical users to trigger extraction jobs and manage tasks via a graphical interface or RE

This project is a specialized TikTok API scraper and data extractor. It functions as a proxy-based web scraper designed to collect user metadata, video posts, and trend feeds, while providing a webhook data pipeline to route scraped information to external URLs via HTTP requests. The tool includes a watermark-free video downloader that saves high-definition content to local storage. It employs cryptographic request signing for server authentication and utilizes session cookie authentication combined with proxy rotation to manage network traffic and avoid rate limits. Capabilities cover bulk

Crawlee-python is a web crawling framework for building scalable scrapers using Python. It serves as a comprehensive tool for web scraping automation, providing a system to extract structured data from websites using both lightweight HTTP requests and headless browser automation. The framework is distinguished by its anti-bot evasion capabilities, which include browser fingerprint impersonation and tiered proxy rotation to bypass detection systems and solve challenges such as Cloudflare. It also incorporates artificial intelligence for autonomous website navigation and schema-based data extra

CyberScraper-2077 is an AI-powered web scraping tool that uses large language models to extract and structure data from websites into organized formats. It functions as an LLM web scraper and AI content parser, transforming unstructured raw web text into specific data schemas. The project distinguishes itself through a suite of anonymity and evasion tools, including proxy rotation, SOCKS-based identity masking, and the ability to route traffic through the Tor network to access hidden onion services. It further includes a bot detection bypass system that employs stealth parameters and custom n

Hardseed is a command-line forum media scraper and content archiver. It extracts images and torrent seeds from internet forums in bulk, saving the media and associated metadata to local directories. The tool utilizes a proxy-enabled web scraping engine that rotates network traffic through multiple proxy servers and protocols to bypass rate limits and network restrictions. It includes a keyword-based media filter that matches user-defined strings within entry titles to include or exclude specific topics. The system manages data extraction through batch-processed thread iteration and range-lim

node-crawler is a programmable web crawler for Node.js that manages request queues and automates data extraction. It functions as a rate-limited HTTP client and a headless HTML parser, providing the infrastructure to visit large sets of URLs asynchronously while preventing duplicate processing through task deduplication. The project distinguishes itself through a proxy rotation manager that cycles user agents and proxy servers to bypass access restrictions. It utilizes the HTTP/2 protocol to improve request performance and server compatibility during large-scale scraping operations. The syst

Crawlee is a web scraping framework designed for building scalable, reliable, and distributed data extraction pipelines. It provides a unified interface for managing headless browser automation and lightweight HTTP requests, allowing developers to handle complex web navigation, dynamic content rendering, and large-scale data collection within a single, modular architecture. The project distinguishes itself through its resource-aware concurrency controller, which dynamically scales task execution based on real-time CPU and memory usage to prevent host machine exhaustion. It also features a rob

ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability. The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a p

This project is a proxy aggregation platform designed to collect and verify free proxy server lists from web platforms, social media, and public repositories. It functions as a crawler framework that gathers proxy data and subscription links, a validation tool for testing server liveness, and a synchronization service for distributing the results. The system uses a plugin-based architecture that allows for the integration of custom Python scripts to handle diverse web source structures. It also includes utilities to transform raw proxy data into standardized configuration formats compatible w

This project is a repository of proxy server addresses, providing a regularly updated dataset of HTTP, SOCKS4, and SOCKS5 endpoints. It functions as a directory for applications that require managed network routing, offering curated connection details to facilitate traffic management and infrastructure tasks. The system distinguishes itself through an automated harvesting pipeline that refreshes the proxy list on an hourly schedule. This process includes geospatial metadata enrichment, which appends location data to each entry, allowing for regional routing and the simulation of local user ac

Translumo is an optical character recognition screen translator and multi-engine orchestrator. It extracts text from active application windows in real time to translate content into different languages, facilitating the localization of software that lacks official translation options. The system distinguishes itself by combining results from several recognition engines and using machine learning to determine the most accurate text extraction. It also functions as a proxy rotating gateway, cycling through IP addresses to prevent translation services from blocking high-volume requests. The pr

musicdl is a command line music downloader and library manager designed for searching and downloading audio tracks and playlists from streaming platforms to local storage. It functions as a tool for music library archiving, allowing for the bulk acquisition of media and the organization of local audio collections. The project includes an AI lyric transcriber that uses machine learning models to generate text lyrics from audio files, supporting synchronized playback where lyrics are highlighted based on playback timestamps. To maintain access to streaming platforms, it employs a network proxy

Colly is a high-performance web scraping framework designed for the automated extraction of structured data from websites. It provides a programmable toolkit that manages the complexities of large-scale data collection, including concurrent request orchestration, automatic cookie handling, and robots.txt compliance. By utilizing an asynchronous execution model, the engine maintains high throughput while preventing resource exhaustion during recursive or distributed crawling tasks. The framework is distinguished by its modular, event-driven architecture, which allows developers to hook into sp

SagerNet is an Android VPN client that routes device traffic through proxy protocols to bypass internet censorship and access restricted content. It functions as a censorship circumvention tool, distributing anti-censorship platforms and helping users evade network restrictions, while also serving as a platform for studying internet censorship patterns through scientific research. The application routes network traffic through proxy protocols by hooking into the Android VpnService API to create a local VPN tunnel, running persistently as a foreground service with a notification to prevent the

Data engineering practice repository providing tutorials, distributed processing engines, and Python data pipeline automation scripts. The system encompasses automated data validation, distributed compute aggregation, embedded columnar querying, lazy evaluation planning, partitioned storage export, and cloud storage retrieval. The capability surface covers cloud integration and storage, data engineering and pipelines, data processing and analytics, data quality and testing, database and storage, file management, and monitoring and observability.

Guardrails is a Python SDK that wraps calls to large language models with configurable validation pipelines, corrective actions, and structured output generation. It provides a unified API layer that connects to over 100 language models, applying consistent validation, streaming, and error-handling across providers. The framework validates and corrects model responses against safety and quality rules, detecting and mitigating risks in both inputs and outputs using pre-built and custom validators. The project distinguishes itself through a validator-pipeline architecture that sequentially appl

Obscura is a web scraping infrastructure and headless browser server designed for AI agents. It provides a system for AI models to control browser sessions, interact with websites, and extract web data using a WebSocket implementation of the Chrome DevTools Protocol. The project focuses on bot detection evasion by randomizing browser fingerprints, masking native functions, and blocking tracking scripts to mimic human behavior. It further secures identities through a traffic layer that routes network requests via HTTP or SOCKS5 proxies. The system supports large-scale data extraction through

req is a chainable HTTP client library for Go designed to simplify request configuration and automatic response decoding into structures. It provides a fluent-interface request builder that allows developers to incrementally define request properties and encapsulate HTTP logic into reusable API SDKs. The project distinguishes itself with a TLS fingerprint emulator that mimics browser network signatures to bypass bot detection and crawler filters. It also includes a concurrent file downloader that increases transfer speeds by fetching large remote files in parallel segments. The library cover

This is a collection of Python scripts designed for extracting data from popular Chinese websites and mobile applications. It functions as a multi-platform data extraction toolkit, capable of automating tasks such as downloading videos from platforms like Bilibili and Douyin, scraping product reviews and images from e-commerce sites like Taobao and JD.com, and booking train tickets on the 12306 railway system. The project distinguishes itself through its focus on automating specific, high-value tasks within the Chinese internet ecosystem. It includes capabilities for solving Chinese CAPTCHA c

Scraperr is a self-hosted web scraping and crawling platform designed for extracting structured data from websites using XPath selectors. It functions as a containerized system for managing scraping jobs through a queue and analyzing the resulting content using artificial intelligence. The project differentiates itself through its Kubernetes-native architecture, allowing for scalable deployment and management via package managers. It includes a crawling engine capable of domain-level spidering to discover linked pages and a data analyzer that uses artificial intelligence to query extracted we

Shadowrocket is a proxy client application for mobile devices that functions as a multi-protocol proxy manager and a rule-based traffic router. It acts as a programmable network gateway, utilizing a virtual network interface to route system-level traffic through secure tunnels. The project distinguishes itself through a programmable environment that executes JavaScript scripts and modules to automate DNS resolution and handle complex network request logic. It further provides an HTTPS traffic inspector capable of decrypting encrypted traffic using custom certificates to modify headers and rew