Awesome GitHub RepositoriesConsistent Hashing

This project is a comprehensive educational resource focused on the principles, patterns, and trade-offs required to design scalable, reliable, and high-performance distributed systems. It provides a structured curriculum that covers the fundamental architectural strategies necessary for building modern software infrastructure, ranging from high-level system decomposition to low-level networking and data management. The repository distinguishes itself by offering deep dives into complex architectural patterns, such as microservices-based decomposition, event-driven communication, and command-

Rustfs is a distributed object storage system designed for high availability and horizontal scalability. It functions as a cluster-based platform that manages data across multiple nodes, providing a self-hosted infrastructure for large-scale storage requirements. The system is built to be container-native, utilizing an operator to automate deployment and management within orchestrated environments. It provides compatibility with standard object storage protocols, allowing existing applications and tools to interact with the storage layer through a translation interface. To ensure long-term re

Pingora is a Rust-based framework for building high-performance network services, including HTTP reverse proxies, layer seven load balancers, and TLS termination proxies. It serves as an asynchronous network library designed to intercept and route HTTP, gRPC, and WebSocket traffic between clients and upstream backend servers. The project enables zero-downtime service updates by handing over listening sockets between processes during binary or configuration upgrades. It utilizes a programmable multi-phase pipeline to modify request and response bodies and headers, and it provides a pluggable T

Goreplay is an HTTP traffic mirroring tool designed to capture live network traffic from production environments and replay it into test systems for validation. It includes a specialized Kubernetes traffic capturer that operates as a daemonset to mirror traffic from specific pods using label selectors and namespace filters, alongside a TCP traffic recorder for intercepting raw network packets. The project features a Kafka traffic pipeline for streaming captured payloads to topics or ingesting messages for playback, and an HTTP request transformer to mask sensitive data or rewrite headers and

Memcached is a high-performance, distributed, in-memory key-value storage and request routing engine. It functions as a volatile data store designed to accelerate dynamic applications by caching objects in RAM, thereby reducing backend database load and providing sub-millisecond response times. The system utilizes a specialized architecture that organizes memory into fixed-size slabs to minimize fragmentation and maximize throughput for high-concurrency workloads. The project distinguishes itself through a multi-threaded, lock-friendly design that scales across CPU cores and supports complex

Buildkit is a programmable container build toolkit and OCI container image builder that converts build definitions into concurrent dependency graphs for image construction. It functions as an OCI image distribution engine, capable of generating container images and exporting artifacts to local storage or remote registries. The project is distinguished by its use of a low-level binary intermediate representation to decouple high-level build languages from the execution engine. It supports multi-platform image builds through user-mode architecture emulation and provides a distributed build cach

Cassandra is a distributed NoSQL database and wide-column store designed for high availability and linear scalability. It functions as a fault-tolerant distributed system that utilizes an LSM-tree storage engine to optimize write throughput and manage massive datasets. The system is a CQL-compliant database, using a structured query language to manage and retrieve tabular data stored across multiple nodes. It organizes information into rows and columns based on a flexible schema and primary keys. The project provides capabilities for horizontal database scaling, distributed data partitioning

Alluxio is a virtual distributed file system and data orchestration layer that serves as a high-performance caching layer between cloud storage and compute clusters. It acts as a distributed data cache designed to accelerate data access for large-scale analytics and machine learning workloads. The system provides a unified interface that presents multiple heterogeneous storage backends as a single coherent namespace. This allows for the unification of diverse storage systems, enabling computation engines to access data from different providers without changing application code. The project c

JanusGraph is a distributed, elastically scalable graph database designed to store and query highly connected data across a cluster of machines. It supports the property graph data model with ACID consistency and integrates multi-model search capabilities including geo, numeric range, and full-text queries. The database also includes a Graph OLAP engine for running batch analytics and global graph computations on large datasets using the Hadoop framework. The project distinguishes itself through a masterless cluster architecture that eliminates single points of failure, allowing every node to

Katran ist ein eBPF-basierter Layer-4-Load-Balancer für hochperformantes Netzwerk-Paket-Forwarding direkt im Linux-Kernel. Er nutzt einen XDP-Netzwerk-Paketprozessor, um den Standard-Kernel-Netzwerk-Stack zu umgehen, was die Latenz minimiert und den Durchsatz für eingehenden Datenverkehr maximiert. Das System implementiert gewichtetes Consistent Hashing, um Netzwerklasten zu verteilen und die Session-Persistenz über Backend-Server hinweg aufrechtzuerhalten. Es optimiert den Rückweg durch ein Direct-Server-Return-Gateway, das es Backends ermöglicht, direkt an Clients zu antworten und den Load-Balancer aus dem ausgehenden Datenverkehrsfluss zu entfernen. Zu den Funktionen gehören lockfreies Connection-Tracking für Session-Stabilität und NUMA-bewusstes Core-Binding zur Eliminierung von Cross-Node-Latenz. Das Projekt unterstützt zudem Backend-Traffic-Weighting, Server-Draining und XDP-Programm-Chaining zur Ausführung von Firewall-Logik vor dem Load-Balancing. Die Implementierung bietet Observability-Tools für die Erfassung von Netzwerkverkehr, Healthcheck-Probe-Routing und gekapselte Traffic-Filterung.

This project is an Android device farm management platform and mobile device testing lab. It provides a web-based interface for remotely controlling, debugging, and managing fleets of physical Android devices. The platform enables remote hardware access through an ADB remote control system, allowing for remote shell execution, application installation via drag-and-drop, and real-time screen interaction through a browser. It includes tools for mapping remote device connections to local ports, enabling the use of integrated development environments for debugging. The system covers hardware inv

Olric is a distributed data grid and in-memory key-value store that partitions and replicates data across a cluster of servers. It serves as a shared memory system for managing distributed maps, performing atomic operations, and acting as an in-memory data cache. The system provides a distributed locking mechanism for concurrency control and a pub-sub messaging system that broadcasts and routes messages over named channels across the cluster. The platform covers wide-ranging capabilities including cluster management and orchestration, data replication with configurable quorums, and automated

Alloy is a clustered telemetry collector and observability data pipeline that functions as an OpenTelemetry collector distribution. It acts as a declarative configuration engine for collecting and routing metrics, logs, traces, and profiles from various sources to monitoring backends. The system distinguishes itself through a distributed architecture that uses consistent hashing to balance scraping targets and collection workloads across multiple nodes. It manages fleet-wide settings via remote configuration fetching and a modular system for importing reusable pipeline patterns. As a Kubernet