Awesome GitHub RepositoriesDistributed Object Stores

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

SeaweedFS is a distributed object store and high-performance file system designed to manage massive volumes of unstructured data. It utilizes a decoupled architecture that separates metadata management from raw data storage, allowing for independent scalability and the efficient handling of billions of files. By providing a POSIX-compliant interface, it enables applications to interact with a unified namespace while maintaining the performance characteristics of a distributed object store. The system distinguishes itself through a multi-region data fabric that supports active-active replicati

FastDFS is a distributed file system and object store designed as a high-capacity file server. It functions as a cluster storage manager that saves, syncs, and accesses large volumes of unstructured data across a network of distributed servers. The system uses unique identifiers for file retrieval and indexing instead of traditional hierarchical naming to avoid metadata bottlenecks. It manages file attributes through key-value metadata mapping and employs a distributed replication model to ensure high availability and data redundancy across storage groups. The project provides capabilities f

CubeFS 是一个分布式云存储系统，旨在管理跨数据中心和混合云的文件和对象存储。它作为一个多租户分布式文件系统和对象存储，能够处理艾字节（exabyte）规模的数据，并利用分布式架构存储非结构化内容。 该系统以其多协议接口层为特色，允许通过 S3、POSIX 和 HDFS 接口同时访问数据。它采用存算分离架构以独立扩展处理和持久化能力，并实施细粒度的隔离策略以分离不同租户间的资源和数据。 可靠性通过可配置的冗余策略进行管理，包括多副本镜像和纠删码。该平台包含一个多级缓存系统以加速数据访问，并与 Kubernetes 通过容器存储接口（CSI）驱动程序集成，以实现持久卷的自动化配置。

Mooncake 是一个用于高性能推理基础设施的存算分离大语言模型服务平台和分布式键值存储系统。它作为一个 GPU 内存编排器和 KV 缓存管理系统，通过在集群间池化和传输键值缓存来加速推理。 该系统的独特之处在于将推理的预填充（prefill）和解码（decode）阶段分离到不同的硬件集群中，以优化资源利用率。它利用高性能 RDMA 分布式缓存和零拷贝传输在计算节点间移动数据，绕过 CPU 以减少延迟和开销。 该平台涵盖了广泛的功能领域，包括分布式内存池化、通过 CXL 进行加速器内存路由，以及向 SSD 进行多级存储卸载。它通过元数据协调服务管理集群状态，并利用基于租约的对象保护和基于水位线的缓存驱逐机制实现资源治理。 该软件已打包用于容器化部署，支持主机网络和硬件设备映射。

FastStream is an asyncio message broker framework for building event-driven applications in Python. It provides a unified interface and a multi-broker messaging abstraction layer that translates generic producer and consumer calls into broker-specific APIs. The framework features a built-in dependency injection container and uses decorators to route messages to asynchronous handler functions. It includes a documentation generator that extracts channel definitions and message formats from code to produce standardized AsyncAPI specifications. The project supports integration with Kafka, Rabbit