Awesome GitHub RepositoriesErasure Coding

MinIO is a software-defined, cloud-native object storage server designed to manage large volumes of unstructured data. It functions as a distributed storage cluster that aggregates multiple independent nodes into a unified, scalable pool, providing a high-performance infrastructure compatible with standard cloud storage protocols and application programming interfaces. The system utilizes a shared-nothing architecture that eliminates central metadata servers, relying instead on a decentralized hash table to map objects across the cluster. Data availability and resilience are maintained throug

Rustfs is a distributed object storage system designed for high availability and horizontal scalability. It functions as a cluster-based platform that manages data across multiple nodes, providing a self-hosted infrastructure for large-scale storage requirements. The system is built to be container-native, utilizing an operator to automate deployment and management within orchestrated environments. It provides compatibility with standard object storage protocols, allowing existing applications and tools to interact with the storage layer through a translation interface. To ensure long-term re

Ceph is a unified, software-defined storage platform designed to provide object, block, and file storage services from a single distributed cluster. By decoupling data management from physical hardware, it enables elastic scaling across commodity hardware, allowing organizations to build large-scale storage infrastructure without reliance on proprietary vendor equipment. The system distinguishes itself through a shared-nothing, distributed architecture that utilizes deterministic hashing for data placement. This approach eliminates centralized metadata bottlenecks, allowing the cluster to sca

CubeFS 是一个分布式云存储系统，旨在管理跨数据中心和混合云的文件和对象存储。它作为一个多租户分布式文件系统和对象存储，能够处理艾字节（exabyte）规模的数据，并利用分布式架构存储非结构化内容。 该系统以其多协议接口层为特色，允许通过 S3、POSIX 和 HDFS 接口同时访问数据。它采用存算分离架构以独立扩展处理和持久化能力，并实施细粒度的隔离策略以分离不同租户间的资源和数据。 可靠性通过可配置的冗余策略进行管理，包括多副本镜像和纠删码。该平台包含一个多级缓存系统以加速数据访问，并与 Kubernetes 通过容器存储接口（CSI）驱动程序集成，以实现持久卷的自动化配置。

YDB 是一个分布式 SQL 数据库和分析引擎，专为水平扩展和强一致性而设计。它作为一个多模型系统，通过提供可序列化 ACID 事务的分布式架构支持事务和分析工作负载。 该系统以其广泛的协议兼容性而著称，实现了用于标准 SQL 驱动程序的 PostgreSQL 有线协议和用于消息传递与流处理的 Kafka 协议。它进一步作为向量数据库，支持用于语义搜索和嵌入的向量索引以及近似最近邻搜索。 该平台使用具有行式和列式格式的混合存储模型管理数据，利用向量化查询执行进行 PB 级分析。其操作范围包括变更数据捕获流、精确一次（exactly-once）持久队列和多区域高可用性。 部署和生命周期管理通过 Kubernetes Operator 和基础设施即代码（IaC）配置提供支持。