Awesome GitHub RepositoriesAtomic Write Coordinators

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

Vitess is a distributed MySQL orchestrator and clustering system designed for horizontal database scaling. It functions as sharding middleware that distributes data and load across multiple MySQL instances to handle growth beyond the capacity of a single machine. The system provides a proxy layer that abstracts data distribution, allowing applications to query a cluster as a single logical database without knowing the physical location of the data. This is achieved through a routing mechanism that intercepts queries and directs them to the appropriate shards based on keyspace mappings. The p

Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity. The project utilizes a standardized REST catalog interface to manage table metadata, ensuring interoperability between different compute engines. This allows diverse query engines to connect to a single table interface and maintain consistency across different processing frameworks. Its core capabilities include managing large-scale analytic tables, coordin

goleveldb est une base de données de stockage clé-valeur embarquée pour Go. Elle fournit une persistance et une indexation des données locales, permettant aux applications de stocker et de récupérer des informations en utilisant des clés uniques sans nécessiter de serveur séparé. La base de données organise les données en utilisant un arbre de fusion structuré en journal (LSM-tree) et une indexation persistante dans l'ordre lexicographique. Cette structure prend en charge les scans de plage efficaces et les recherches basées sur les préfixes. Le système inclut des capacités pour les écritures par lots atomiques afin d'assurer la cohérence des données et d'éviter les mises à jour partielles. Les performances sont gérées via le write-ahead logging, la mise en mémoire tampon et les filtres de Bloom pour réduire les entrées/sorties disque inutiles.

pgdog is a PostgreSQL sharding proxy, distributed SQL router, and connection pooler. It is designed to enable horizontal data distribution by splitting tables and indices across multiple independent servers to scale storage and processing capacity. The project distinguishes itself through online resharding capabilities, using logical replication to move data between shards without application downtime. It supports multiple routing strategies, including hash, list, and range-based query routing, and manages distributed atomic transactions using a two-phase commit process to ensure consistency