Awesome GitHub RepositoriesDistributed Data Replication

FoundationDB is an ACID-compliant distributed transactional key-value store. It functions as a scalable database engine that ensures strict serializability and data consistency across a cluster of servers using a shared-nothing architecture. The system is distinguished by its multi-region replication capabilities, allowing data to be synchronized across different datacenters for high availability and disaster recovery. It utilizes optimistic concurrency control to manage distributed transactions and employs a majority-based coordination system to maintain cluster state. The platform provides

Rook is a Kubernetes storage orchestrator and distributed storage operator that automates the deployment and management of storage clusters. It serves as a multi-protocol storage provider, offering block, file, and object storage capabilities to containerized workloads. The system focuses on providing a self-healing storage cluster that replicates data across hardware nodes to maintain availability and recover from failures. It uses an operator-led model to handle the installation, scaling, and upgrades of storage nodes and daemons. The orchestrator covers a broad range of provisioning servi

FastDFS is a distributed file system and object store designed as a high-capacity file server. It functions as a cluster storage manager that saves, syncs, and accesses large volumes of unstructured data across a network of distributed servers. The system uses unique identifiers for file retrieval and indexing instead of traditional hierarchical naming to avoid metadata bottlenecks. It manages file attributes through key-value metadata mapping and employs a distributed replication model to ensure high availability and data redundancy across storage groups. The project provides capabilities f

Hazelcast is a distributed data platform that combines an in-memory data grid with a stream processing engine to support real-time analytics and event-driven applications. It functions as a partitioned, distributed key-value store that replicates data across cluster nodes to provide low-latency access and high availability. The platform also serves as a distributed SQL query engine, allowing users to execute standard SQL statements against both in-memory datasets and external data sources. What distinguishes Hazelcast is its use of a distributed consensus subsystem to maintain strongly consis

This is an open-source, crowd-sourced wiki textbook that teaches Linux system programming in C. It covers the core operating system concepts of process management through the fork-exec-wait model, dynamic memory allocation using implicit free list heap allocators, inode-based file systems, inter-process communication via pipes and shared memory, POSIX threads with synchronization primitives, signal-based asynchronous notification, virtual memory with page table translation, and runtime diagnostics using Valgrind and GDB. The textbook distinguishes itself by providing practical, implementation

Fluvio is a distributed event streaming platform and cloud-native streaming engine designed for collecting, persisting, and replicating real-time data streams across a distributed cluster. It functions as a real-time data pipeline for building stateful workflows that ingest, enrich, and export data between external sources and sinks. The platform is distinguished by its use of WebAssembly to execute compiled modules for in-line data transformations and filtering. This allows for the execution of custom business logic to reshape information in motion without requiring a restart of the cluster.

GlusterFS 是一个软件定义的分布式文件系统和横向扩展存储集群，将来自多个服务器的磁盘资源聚合到一个单一的全局命名空间中。它作为一个统一的存储平台，允许通过文件、块和对象存储接口暴露相同的基础数据。 该系统通过去中心化架构脱颖而出，该架构使用一致性哈希在网络节点之间分发文件，而无需中央元数据服务器。它通过自愈复制、基于仲裁的一致性来防止脑裂场景，以及用于跨地理位置灾难恢复的异步地理复制，确保了数据完整性和可用性。 该平台通过 NFS、Samba 和 iSCSI 提供广泛的多协议存储导出功能，以及全面的卷管理功能，包括时间点快照、存储配额和弹性容量扩展。安全性通过传输中的 TLS 加密、身份管理集成和细粒度的 POSIX 访问控制来处理。 该软件可通过社区仓库以二进制包的形式提供，适用于各种 Linux 发行版。

m3 是一个分布式时间序列数据库，专为高分辨率指标和高基数数据管理而设计。它作为一个可扩展的存储系统和多集群查询引擎，提供了一个分布式指标聚合器，能够在数据提交到存储之前进行降采样和汇总。 该项目以其使用 etcd 进行节点成员管理和分片放置的协调集群模型而脱颖而出。它支持多种摄取协议，包括 Prometheus 远程写入协议、InfluxDB 行协议和 Graphite Carbon 纯文本协议，并提供与 PromQL 和 Graphite 兼容的查询接口。 该系统涵盖了广泛的功能领域，包括列式时间序列存储、同步数据复制和分布式查询扇出。它集成了数据生命周期自动化、基于法定人数 (Quorum) 的一致性调整，以及基于标签的序列索引，以在隔离的命名空间中保持数据完整性和检索速度。 集群编排和组件放置通过自动化工具和 Operator 进行管理，以确保高可用性和均衡的数据分布。

Iggy 是一个分布式消息流平台和多协议消息代理，作为持久化分布式日志存储运行。它为使用追加日志（append-only log）发布和消费二进制消息提供了基础设施，通过 Viewstamped Replication 确保跨节点的高可用性和数据一致性。 该平台以其专业的 LLM 流式基础设施而著称，该基础设施使用服务器协议将大语言模型连接到流数据和系统控制。这包括通过 HTTP 或标准输入输出进行上下文管理和数据桥接的标准化协议。 该系统涵盖了广泛的功能，包括带有模块化源和汇插件的数据管道编排、用于水平扩展的消费者组协调，以及跨 TCP、QUIC、HTTP 和 WebSocket 的多协议传输支持。它还集成了诸如用于静态和传输中数据的 AES-256-GCM 加密等安全原语，并通过 Prometheus 指标、OpenTelemetry 追踪和操作 Web 仪表板提供可观测性。 该服务器可以使用容器镜像部署，并通过 Kubernetes 进行编排。

Dynomite 是一个分布式数据分片层和键值存储引擎代理。它作为一个分片和跨多个节点复制数据的分发层，将单服务器数据存储转换为可扩展的点对点系统。 该系统充当多数据中心数据复制器，在不同地理位置之间同步数据，以确保站点故障期间的弹性和高可用性。它管理键值数据的分发，以实现线性数据存储扩展和冗余存储。 该项目提供了存储引擎分片和高可用性网络的能力。它将传入请求路由到本地或远程存储引擎，同时维护通信协议，并通过加密保护节点间通信。

go-fastdfs 是一个专为构建私有云存储而设计的分布式文件系统和对象存储服务器。它提供了一个兼容 FastDFS 的存储实现，管理存储节点集群以处理大规模文件上传和下载。 该系统通过去中心化架构实现高可用性，无需中央协调器即可自动同步数据并修复多台机器上的故障。它特别支持通过 HTTP 进行断点续传，允许大文件传输在网络不稳定时从最后一个成功字节处暂停并恢复。 核心功能包括通过基于 SHA1 的内容去重进行存储资源优化，以及合并小文件以减少文件系统 inode 消耗。该项目还集成了一个图像处理流水线，可在下载过程中执行动态缩放和调整大小，并使用基于令牌的认证来保护文件访问。 该系统可通过 Docker 容器部署。

Olric is a distributed data grid and in-memory key-value store that partitions and replicates data across a cluster of servers. It serves as a shared memory system for managing distributed maps, performing atomic operations, and acting as an in-memory data cache. The system provides a distributed locking mechanism for concurrency control and a pub-sub messaging system that broadcasts and routes messages over named channels across the cluster. The platform covers wide-ranging capabilities including cluster management and orchestration, data replication with configurable quorums, and automated

Garage is a distributed object storage system that provides an S3-compatible API gateway. It is designed to synchronize metadata across distributed nodes using conflict-free replicated data types and Merkle-tree state alignment to maintain cluster-wide consistency. The system ensures data resilience through zone-aware replication, distributing data copies across multiple physical locations. It employs quorum-based request routing and versioned layout management to validate and commit cluster configuration changes. The project covers a broad range of operational capabilities, including automa