How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
Distribute crawler is a distributed web scraping framework that integrates with Scrapy to coordinate multiple crawler instances across clusters. It utilizes a centralized task queue to manage and scale concurrent data collection operations, enabling horizontal scaling of scraping tasks across multiple worker nodes. The framework distinguishes itself through its focus on large-scale data management and traffic control. It persists scraped items and binary assets into document-oriented database clusters, utilizing deduplication logic to optimize bandwidth and storage. To maintain consistent dat
ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple sources through a web API. At its core, it runs scheduled background processes that scrape free and paid proxy websites, test each proxy's availability against configurable target URLs using asynchronous HTTP clients, and store the results in a Redis-backed sorted set where proxies are scored and ranked by reliability. The system distinguishes itself through a pluggable crawler architecture that allows users to add new proxy sources by writing a simple class with target URLs and a p
Garage is a distributed object storage system that provides an S3-compatible API gateway. It is designed to synchronize metadata across distributed nodes using conflict-free replicated data types and Merkle-tree state alignment to maintain cluster-wide consistency. The system ensures data resilience through zone-aware replication, distributing data copies across multiple physical locations. It employs quorum-based request routing and versioned layout management to validate and commit cluster configuration changes. The project covers a broad range of operational capabilities, including automa
jx is a GitOps delivery platform and Kubernetes CI/CD orchestrator designed to automate the building and deployment of applications. It functions as a cloud native pipeline manager that executes container-based build and deployment sequences using a catalog of reusable tasks. The project distinguishes itself through the automated orchestration of preview environments, which are created and destroyed based on pull request activity to enable validation before merging. It employs a GitOps-based state synchronization model to maintain the desired state of clusters by polling git repositories and
Haipproxy is a high-availability proxy gateway and distributed proxy pool manager. It consists of a system for storing and rotating verified IP proxy addresses using Redis, a web crawling system to discover anonymous proxies from public sources, and a validation engine that checks proxy functionality against specific target domains.
The main features of spiderclub/haipproxy are: Proxy and User-Agent Rotation Middleware, Gateway High Availability, Distributed Proxy Managers, Redis-Backed Proxy Pools, Local Middleware Proxies, Local Proxy Bridge Configurations, Proxy Pool Caches, Proxy Crawler Frameworks.
Open-source alternatives to spiderclub/haipproxy include: gnemoug/distribute_crawler — Distribute crawler is a distributed web scraping framework that integrates with Scrapy to coordinate multiple crawler… python3webspider/proxypool — ProxyPool is a proxy pool manager that automatically collects, validates, and serves HTTP proxies from multiple… deuxfleurs-org/garage — Garage is a distributed object storage system that provides an S3-compatible API gateway. It is designed to… ravendb/ravendb — RavenDB is a multi-model NoSQL document database designed for high-performance, ACID-compliant data storage. It… jenkins-x/jx — jx is a GitOps delivery platform and Kubernetes CI/CD orchestrator designed to automate the building and deployment of… hazelcast/hazelcast — Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to…