Awesome GitHub RepositoriesData Write Throughput Optimizers

Presto is a distributed SQL query engine designed for high-performance analytical processing across heterogeneous data sources. It functions as a data federation platform and massively parallel processing engine, allowing users to execute interactive queries against diverse storage systems without requiring data migration. By mapping remote metadata and structures to a unified relational namespace, it enables seamless cross-platform analysis through a standard SQL interface. The engine distinguishes itself through a pluggable connector architecture and a shared-nothing distributed processing

ScyllaDB is a distributed NoSQL database engine designed for high-throughput data storage and low-latency performance at scale. It functions as a shard-aware platform that manages large-scale datasets across distributed clusters, providing a foundation for real-time applications that require consistent availability and operational stability. The system distinguishes itself through a shared-nothing architecture that distributes data across independent CPU cores to eliminate lock contention. It incorporates a user-space networking stack and an asynchronous event-driven engine to maximize hardwa

Metaflow is a Python machine learning framework and MLOps workflow orchestrator designed to manage the lifecycle of data pipelines from local prototyping to production. It serves as a distributed compute manager and an experiment tracking system, enabling the creation of reproducible pipelines that transition between development and high-availability production environments. The framework distinguishes itself through an integrated checkpointing system that automatically persists intermediate data artifacts to remote storage, allowing failed runs to be resumed from the last successful step. It

CubeFS 是一个分布式云存储系统，旨在管理跨数据中心和混合云的文件和对象存储。它作为一个多租户分布式文件系统和对象存储，能够处理艾字节（exabyte）规模的数据，并利用分布式架构存储非结构化内容。 该系统以其多协议接口层为特色，允许通过 S3、POSIX 和 HDFS 接口同时访问数据。它采用存算分离架构以独立扩展处理和持久化能力，并实施细粒度的隔离策略以分离不同租户间的资源和数据。 可靠性通过可配置的冗余策略进行管理，包括多副本镜像和纠删码。该平台包含一个多级缓存系统以加速数据访问，并与 Kubernetes 通过容器存储接口（CSI）驱动程序集成，以实现持久卷的自动化配置。

Mountpoint for Amazon S3 is a FUSE-based filesystem client that mounts S3 buckets as local directories, enabling standard file operations on objects without custom code. It enforces S3 bucket permissions through AWS Identity and Access Management policies on every operation, and implements lazy object materialization to fetch content on-demand rather than downloading entire objects at mount time. The filesystem maps S3's flat key namespace into a hierarchical directory structure using forward slashes as path separators, and supports write-back object assembly that accumulates local writes into

Dragonboat 是一个 Go 语言实现的 Raft 一致性协议，旨在维护分布式节点集群间的一致状态。它提供了一个用于构建分布式状态机的库，确保系统故障期间的数据完整性和容错能力。 该项目通过多组 Raft 实现脱颖而出，它将数据分区到独立的共识组中，以分配工作负载并提高整体系统处理能力。它还结合了双向 TLS 来加密节点间通信并验证集群成员的身份。 该系统包括支持内存和磁盘持久化的高性能状态机功能。它具有读路径优化以确保一致性而不生成新的日志条目，用于自定义日志后端的插件式存储接口，以及用于在节点多数永久丢失后恢复可用性的仲裁恢复管理工具。 通过导出集群健康指标来支持操作稳定性。

OpenTSDB 是一个分布式时间序列数据库和指标引擎，专为存储和管理海量高基数系统指标而设计。它作为一个数据存储和分析平台，支持跨分布式集群的大规模指标摄取和基础设施性能监控。 该系统以其支持 HBase、Cassandra 和 Google Bigtable 等多个后端的分布式存储抽象而著称。它利用分层指标树来组织时间序列，并采用数字标识符索引来减少存储占用并加速标记指标的查找。 该项目涵盖了广泛的能力领域，包括具有分布式百分位数计算和降采样功能的时间序列数据分析，以及全面的元数据管理。它提供用于数据摄取和查询的 API 集成、用于性能优化的堆外缓存，以及用于数据完整性审计和异常分析的工具。 该系统通过用于数据库管理和指标树同步的命令行界面进行管理。

Velox 是一个高性能 C++ 查询执行引擎和列式数据处理库。它作为一个用于实现分析型查询引擎的可组合框架，提供了向量化表达式评估器和数据管理系统工具包。 该项目以使用向量化列式执行和基于 Arena 的内存分配来处理大规模数据集而著称。它具有专门的优化功能，如广播连接表缓存、动态过滤器下推和字典编码，以减少内存开销并加速分析读取。 该引擎涵盖了广泛的分析能力，包括实现哈希连接、合并连接和半连接，以及多阶段并行聚合和窗口函数计算。它提供了用于列式内存存储、Parquet 数据解码以及与云存储集成的原语。 通过用于自定义标量和聚合函数的函数注册系统提供可扩展性，并提供高级绑定以将 C++ 逻辑连接到 Python。

Orioledb 是一个专为 PostgreSQL 设计的云原生存储引擎，旨在替换默认存储层，以提高现代硬件上的垂直扩展性和性能。它作为一个索引组织表存储，直接在主索引内组织表行，以加速数据检索。 该引擎利用撤销日志存储系统来管理数据版本控制，这消除了手动清理（vacuuming）的需要并防止了表膨胀。它还通过块级和页级数据压缩进一步减少了磁盘占用。 该项目提供了高级索引管理和自动化数据库维护功能。它包括通过行级日志记录实现高可用性恢复的功能，以及用于分析空间利用率和验证表完整性的工具。

sofa-jraft is a Java implementation of the Raft consensus algorithm. It serves as a distributed consensus engine and linearizable state machine designed to ensure high availability and data consistency across a cluster of nodes. The project provides a replicated key-value store and a coordination engine for managing distributed state. It distinguishes itself through support for multi-group consensus sharding to distribute traffic and a service provider interface that allows for custom log storage and entry encoding implementations. The system covers a wide range of distributed capabilities,

SlateDB is a cloud-native key-value store and distributed database engine that utilizes a log-structured merge-tree architecture. It serves as a transactional storage layer designed to persist data directly to cloud object storage. The engine differentiates itself by optimizing read performance for remote storage through the use of bloom filters and multi-level block caching. It employs a single-writer multi-reader model and provides the ability to create zero-copy clones via copy-on-write checkpointing. The system supports atomic transactions, range queries, and snapshot-based concurrency c

This project is a reference library of architectural blueprints, study materials, and design patterns for building scalable, high-availability distributed systems. It serves as a technical guide for scalability engineering, providing structural solutions for common engineering challenges. The repository focuses on distributed systems design, covering essential patterns for data replication, consensus algorithms, and transaction management. It distinguishes itself by offering detailed blueprints for specialized domains, including real-time data streaming, large-scale data storage, and high-ava