Awesome GitHub RepositoriesAtomic Write Coordinators

Prefect is a workflow orchestration platform designed to define, schedule, and monitor complex data pipelines as Python code. It functions as a container-native engine that wraps individual tasks in isolated environments, ensuring consistent dependencies and resource allocation across diverse infrastructure. By utilizing a state-machine-based orchestration model, the system tracks execution progress through discrete transitions and persistent event logs to maintain reliable and observable task processing. The platform distinguishes itself through a decoupled worker-API architecture, which sep

Vitess is a distributed MySQL orchestrator and clustering system designed for horizontal database scaling. It functions as sharding middleware that distributes data and load across multiple MySQL instances to handle growth beyond the capacity of a single machine. The system provides a proxy layer that abstracts data distribution, allowing applications to query a cluster as a single logical database without knowing the physical location of the data. This is achieved through a routing mechanism that intercepts queries and directs them to the appropriate shards based on keyspace mappings. The p

Iceberg is an open table format and big data table manager designed for huge analytic datasets in cloud storage. It provides a specification for tracking large-scale datasets to maintain transactional consistency and structural integrity. The project utilizes a standardized REST catalog interface to manage table metadata, ensuring interoperability between different compute engines. This allows diverse query engines to connect to a single table interface and maintain consistency across different processing frameworks. Its core capabilities include managing large-scale analytic tables, coordin

goleveldb هي قاعدة بيانات تخزين مفتاح-قيمة مدمجة للغة Go. توفر استمرارية البيانات المحلية والفهرسة، مما يسمح للتطبيقات بتخزين واسترجاع المعلومات باستخدام مفاتيح فريدة دون الحاجة إلى خادم منفصل. تنظم قاعدة البيانات البيانات باستخدام شجرة دمج منظمة بالسجل (log-structured merge-tree) وفهرسة دائمة بترتيب معجمي. يدعم هذا الهيكل عمليات مسح النطاق الفعالة وعمليات البحث القائمة على البادئة. يتضمن النظام قدرات لعمليات الكتابة المجمعة الذرية لضمان اتساق البيانات وتجنب التحديثات الجزئية. تتم إدارة الأداء من خلال تسجيل الكتابة المسبقة (write-ahead logging)، والتخزين المؤقت القائم على الذاكرة، وفلاتر bloom لتقليل إدخال وإخراج القرص غير الضروري.

pgdog is a PostgreSQL sharding proxy, distributed SQL router, and connection pooler. It is designed to enable horizontal data distribution by splitting tables and indices across multiple independent servers to scale storage and processing capacity. The project distinguishes itself through online resharding capabilities, using logical replication to move data between shards without application downtime. It supports multiple routing strategies, including hash, list, and range-based query routing, and manages distributed atomic transactions using a two-phase commit process to ensure consistency