Awesome GitHub RepositoriesDistributed Cluster Coordination

Storm is a distributed stream processing framework and fault-tolerant compute engine designed for executing real-time continuous computations across a cluster of machines. It functions as a stateful stream processor and cluster topology manager, enabling the deployment and monitoring of distributed data flow configurations. The system ensures exactly-once semantics by utilizing transactional state management to guarantee that every message in a data stream is processed exactly one time. It further operates as a distributed RPC system, allowing for the integration of non-native languages throu

Otter is a distributed database synchronization system and change data capture tool designed to replicate data between databases across multiple geographic regions. It functions as a synchronization orchestrator and ETL data pipeline that mirrors records and associated files in real time. The system employs incremental log parsing to capture database changes and utilizes a consistency-based convergence algorithm and loop-avoidance logic to manage bi-directional replication. It processes data through a pipeline of selection, extraction, transformation, and loading to handle joins and format co

xmall is a distributed e-commerce platform based on a service-oriented architecture. It separates business logic into independent services that communicate over a network to ensure scalability and fault tolerance, utilizing a decoupled storefront interface for customer transactions. The platform employs a distributed architecture using Dubbo for service orchestration and Zookeeper for cluster coordination and service discovery. It integrates a specialized set of components including an asynchronous message broker for background tasks, an indexed search system for product catalogs, and a centr

Storm is a distributed stream processing framework designed to execute unbounded computations across a cluster to process real-time data streams. It functions as a data pipeline orchestrator that allows users to define and deploy declarative data flow graphs connecting streaming sources to processing components. The system operates as a multi-tenant distributed compute engine that isolates workloads and limits resource usage across shared clusters using dedicated pools and access control. It is also a secure distributed processing engine that employs encrypted node communication and SSL-secur

Apache Storm is a distributed stream processing framework and real-time data processing engine. It functions as a fault-tolerant distributed computing system designed to analyze data in motion across a cluster of machines for continuous stream computation. The system enables the creation of fault-tolerant data pipelines and scalable event processing by distributing workloads across a network of computing nodes. This architecture ensures low latency and high throughput for live data while allowing the system to recover automatically from individual node failures. The framework provides capabi

Flashlight is a standalone C++ machine learning library and tensor library used for building and training neural networks. It functions as a comprehensive neural network framework and automatic differentiation engine, providing the tools to construct computation graphs and calculate gradients via backpropagation. The project serves as a distributed training framework, utilizing all-reduce operations to synchronize gradients and parameters across multiple compute nodes and devices. It distinguishes itself through deep integration of high-performance tensor manipulation, native device memory in

Pigsty 是一个全面的数据库基础设施编排平台，旨在自动化高可用 PostgreSQL 集群的全生命周期。它作为一个基础设施即代码（IaC）框架，通过幂等 Playbook 管理集群协调、节点配置与服务发现。通过集成分布式共识机制，该平台确保了在包括裸机与虚拟化基础设施在内的多样化环境中，自动化故障转移与一致的状态强制执行。 该平台通过一套超越标准数据库管理的强大运营能力脱颖而出。它具备内置的观测流水线，将指标、日志与追踪聚合到集中式仪表盘中，用于实时性能监控与诊断分析。此外，它还提供了一个模拟专有线路协议与 SQL 语法的迁移框架，允许将遗留企业数据库工作负载集成到现代关系型环境中。 该系统涵盖了广泛的功能面，包括带有写时复制（CoW）克隆以实现快速部署的高级存储管理，以及协调关系型引擎与缓存及对象存储服务的多数据库编排。它还整合了安全加固、自动化备份与恢复，以及通过分层代理进行的流量路由，以将客户端连接与底层集群拓扑解耦。 该项目以自包含的包镜像模型分发，能够在安全或离线环境中实现一致的部署与依赖管理。

m3 是一个分布式时间序列数据库，专为高分辨率指标和高基数数据管理而设计。它作为一个可扩展的存储系统和多集群查询引擎，提供了一个分布式指标聚合器，能够在数据提交到存储之前进行降采样和汇总。 该项目以其使用 etcd 进行节点成员管理和分片放置的协调集群模型而脱颖而出。它支持多种摄取协议，包括 Prometheus 远程写入协议、InfluxDB 行协议和 Graphite Carbon 纯文本协议，并提供与 PromQL 和 Graphite 兼容的查询接口。 该系统涵盖了广泛的功能领域，包括列式时间序列存储、同步数据复制和分布式查询扇出。它集成了数据生命周期自动化、基于法定人数 (Quorum) 的一致性调整，以及基于标签的序列索引，以在隔离的命名空间中保持数据完整性和检索速度。 集群编排和组件放置通过自动化工具和 Operator 进行管理，以确保高可用性和均衡的数据分布。

Lunatic 是一个 WebAssembly 运行时和并发进程管理器，实现了 Erlang 风格的轻量级并发和容错模型。它作为一个分布式 Actor 系统运行，其中隔离的进程通过跨链接节点网络的异步消息传递进行通信。 该系统利用 WebAssembly 沙箱环境来隔离内存并限制每个单独进程的系统调用权限。这种基于能力的安全模型确保进程被沙箱化，以安全地执行不受信任的代码。 该平台提供用于分层监控和失败进程自动重启的容错监督树。它使用抢占式工作窃取调度程序来执行数千个轻量级绿色线程，从而管理高并发工作负载。 该运行时支持分布式系统协调，通过跨不同物理机器集群节点，并处理 TCP 和 WebSocket 等协议的网络流量。

Stolon 是一个用于 PostgreSQL 的云原生数据库管理器和编排器。它作为一个高可用管理器，旨在自动化分布式环境中的故障转移、复制和集群管理。 该系统协调同步和异步数据镜像，并使用服务发现机制动态检测数据库节点地址。它通过代理将客户端流量路由到当前主实例，以在角色切换期间保持应用程序访问。 该项目通过时间点恢复和维护多站点备用集群来涵盖灾难恢复。它还包括通过加密保护数据库连接以及自动化检测网络内进程地址更改的功能。

LXD is a system container manager and virtual machine manager that provides a unified interface for running full Linux systems. It acts as a container cluster orchestrator, an image format converter, and an infrastructure manager that exposes control through a REST API and language-specific SDKs. The project distinguishes itself by providing a unified container and virtual machine abstraction, treating both as generic instances within a single management layer. It supports distributed cluster coordination to synchronize state and distribute workloads across multiple physical nodes. The syste

该项目是一个基于 Spring Cloud 微服务平台的企业级多租户分布式系统架构。它提供了一套完整的微服务管理框架，重点在于多租户数据架构和集中式身份认证服务。 该平台通过集成 OAuth2 身份提供商来管理单点登录（SSO）、基于角色的访问控制（RBAC）以及分布式服务间的 JWT 令牌颁发，从而在身份与安全方面表现出色。此外，它通过多租户数据隔离实现了组织边界的划分，确保不同租户间的资源和数据在逻辑或物理上相互独立。 系统涵盖了广泛的分布式能力，包括通过 API 网关路由和熔断机制实现的服务治理，以及通过分布式事务和锁机制实现的数据协调。它还包含用于请求追踪和集中式日志记录的分布式可观测性栈，以及实时搜索引擎同步和异步事件驱动的消息传递机制。 开发工作流通过应用程序代码生成和平台特定的二进制打包自动化工具提供支持。

esProc 是一个分布式 ETL 框架和嵌入式数据计算引擎。它为 Java 虚拟机提供了一种结构化数据语言，专为关系查询、复杂数据计算和结构化数据分析而设计。 该系统具有自然语言数据查询界面，利用大语言模型将请求转换为针对结构化数据集的可执行查询。它采用具有简洁语法的领域特定查询语言来建立表关系并检索信息。 该平台涵盖跨不同关系型和 NoSQL 源的数据集成，并管理 ETL 工作流以在文件和数据库之间移动数据。其他功能包括结构化数据报告生成、用于分步执行可视化的实时网格界面，以及集成自定义外部共享库的能力。

FreeSql 是一个 .NET 对象关系映射器（ORM）和数据访问层，可将面向对象的代码转换为适用于多种关系型数据库提供程序的 SQL。它作为一个流畅的 SQL 查询构建器和数据库架构同步器，允许开发者将数据库表和索引结构与实体类定义保持一致。 该框架专门针对 .NET Native AOT 进行了优化，以确保更小的内存占用和更快的启动时间。它包含一个数据库流量管理器，通过读写分离、动态分表和基于租户的数据隔离来分配负载。 其广泛的功能包括使用特定于提供程序的批量复制机制实现高性能数据摄入，利用窗口函数和递归 CTE 进行高级查询，以及基于 AOP 的数据变更审计监控。该系统还提供用于自动迁移的架构管理工具，以及用于从数据库元数据生成实体类的开发实用程序。

InternetArchitect 是一个教育性文档和源代码合集，旨在作为高并发架构课程。它作为一个分布式系统实现指南，提供技术模式和实践示例，用于设计在重负载下保持稳定性的可扩展互联网架构。 该项目专注于高性能数据库优化和微服务设计模式。它涵盖了通过数据库分片和代理层减少延迟并提高吞吐量的策略，以及在分布式集群中协调全局状态的方法。 架构范围包括用于加速数据检索的多级缓存策略，以及用于管理解耦微服务之间通信的服务发现框架。它还解决了分布式状态协调问题，并使用负载均衡网格在后端服务器之间分配网络流量。

该项目是一个基于 Markdown 的个人知识库和数字学习日志，用于存储书籍和技术文献的笔记与摘要。它作为一个阅读摘要仓库和技术参考库，用于归档非虚构类和专业材料的关键要点与见解。 该集合作为一个数字花园，用于整理来自书籍、文章和视频的见解。它专门专注于将复杂的架构模式、技术概念和专业领导力原则提炼成可搜索的格式，以供长期参考。 该仓库还通过收集可复用的思维框架、心智模型和专业决策工具来组织个人知识。它包含一个用于通过管理已完成和进行中作品列表来跟踪阅读进度的系统。

FlagAI is a distributed deep learning framework and platform designed for the end-to-end lifecycle of large-scale foundation models. It provides a toolkit for training, fine-tuning, and deploying large language models and multi-modal systems across multi-node computing clusters. The project features hardware-agnostic compute abstractions to ensure consistent execution across different accelerators. It includes a dedicated library for parameter-efficient fine-tuning, allowing large neural networks to be adapted to specific tasks with minimal parameter updates and reduced computational overhead

Maestro is a distributed job scheduler and containerized data pipeline tool designed to coordinate complex sequences of tasks. It functions as a Kubernetes workflow orchestrator and MLOps automation platform, utilizing directed acyclic graphs to manage task dependencies and execution order across computing clusters. The system distinguishes itself through the use of isolated container environments for each workflow step, ensuring consistent runtime dependencies. It incorporates an asynchronous event bus to coordinate state transitions and provides lifecycle hook integration that dispatches sy

Oban is a distributed background job processing system and task scheduler that uses PostgreSQL for transactional job storage and reliable execution across multiple nodes. It serves as a PostgreSQL-backed background worker and job queue, coordinating task execution and concurrency through a relational database to ensure delivery guarantees. The system differentiates itself through a distributed workflow orchestrator capable of managing multi-step processing pipelines, dependent job sequencing, and shared context. It provides advanced orchestration tools including job batching, chunked processi

Nano is a distributed application framework designed for building systems using an actor-based messaging model. It functions as a distributed actor framework that decouples components through asynchronous messaging to maintain state isolation across a server cluster. The system acts as a cluster message dispatcher and session-aware request router, tracking client state to route incoming messages to the specific agent holding the session data. It utilizes a distributed agent registry to coordinate the dispatching of messages between multiple application instances acting as agents. The framewo