Awesome GitHub RepositoriesAtomic Write Coordinators

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

Vitess is a distributed MySQL orchestrator and clustering system designed for horizontal database scaling. It functions as sharding middleware that distributes data and load across multiple MySQL instances to handle growth beyond the capacity of a single machine. The system provides a proxy layer that abstracts data distribution, allowing applications to query a cluster as a single logical database without knowing the physical location of the data. This is achieved through a routing mechanism that intercepts queries and directs them to the appropriate shards based on keyspace mappings. The p

Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity. The project utilizes a standardized REST catalog interface to manage table metadata, ensuring interoperability between different compute engines. This allows diverse query engines to connect to a single table interface and maintain consistency across different processing frameworks. Its core capabilities include managing large-scale analytic tables, coordin

goleveldb 是一个 Go 语言的嵌入式键值存储数据库。它提供本地数据持久化和索引，允许应用使用唯一键存储和检索信息，而无需单独的服务器。 该数据库使用日志结构合并树（LSM-tree）和按字典顺序排列的持久化索引来组织数据。这种结构支持高效的范围扫描和基于前缀的搜索。 该系统包含用于原子批量写入的功能，以确保数据一致性并避免部分更新。性能通过预写日志（WAL）、基于内存的缓冲和布隆过滤器进行管理，以减少不必要的磁盘输入和输出。

pgdog is a PostgreSQL sharding proxy, distributed SQL router, and connection pooler. It is designed to enable horizontal data distribution by splitting tables and indices across multiple independent servers to scale storage and processing capacity. The project distinguishes itself through online resharding capabilities, using logical replication to move data between shards without application downtime. It supports multiple routing strategies, including hash, list, and range-based query routing, and manages distributed atomic transactions using a two-phase commit process to ensure consistency