Awesome GitHub RepositoriesData Checkpointing

Ray is a distributed computing framework designed to scale Python and Java applications across clusters by abstracting task scheduling and resource management. It functions as a resource-aware execution engine that manages task dependencies, placement, and fault tolerance across networked compute nodes. At its core, the system provides a stateful actor model, allowing developers to define classes that run in dedicated processes to maintain and mutate internal state across remote method calls. The framework distinguishes itself through a robust cross-language interoperability layer, enabling f

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

Metaflow is a Python machine learning framework and MLOps workflow orchestrator designed to manage the lifecycle of data pipelines from local prototyping to production. It serves as a distributed compute manager and an experiment tracking system, enabling the creation of reproducible pipelines that transition between development and high-availability production environments. The framework distinguishes itself through an integrated checkpointing system that automatically persists intermediate data artifacts to remote storage, allowing failed runs to be resumed from the last successful step. It

Sui is a blockchain platform featuring an object-centric state model and resource-oriented smart contracts. It utilizes parallel transaction execution to increase network throughput and supports programmable transaction blocks that bundle multiple operations into single atomic units. The platform distinguishes itself with a capability-based access control system and zero-knowledge login mechanisms, enabling users to authenticate via identity providers without seed phrases. It also implements deterministic object addressing to allow predictable state lookups and supports the creation of soulbo

该项目是一个基于 JAX 的 Transformer 框架和大语言模型训练器，专为在 TPU 硬件加速器上构建和训练分布式模型而设计。它提供了一个通过在设备网格上拆分权重和计算来减少内存开销并提高处理速度的系统，用于预训练和微调自回归模型。 该框架包含一个 TPU 计算编排器，用于资源配置和自动化远程分布式节点上的依赖安装。它还具有一个模型权重转换器，能够在不同的硬件配置和数值精度之间转换和重新分片检查点。 该项目涵盖了更广泛的功能，包括用于云存储的分片检查点管理、具有状态恢复的流式数据加载，以及用于模型推理的基于核的文本生成。它进一步支持针对 TPU 和 GPU 集群的 XLA 编译硬件加速，并提供针对标准化语言任务进行性能基准测试的工具。

KurrentDB is an event-native database designed for event sourcing and event-driven architectures. It stores events as immutable, ordered records in streams, preserving a complete audit trail and enabling temporal queries. The database uses gRPC for all client-server and inter-node communication, providing efficient binary serialization and bidirectional streaming, and supports atomic multi-stream writes that ensure consistency across multiple streams in a single transaction. The database distinguishes itself with a built-in JavaScript projection engine that transforms, filters, and aggregates

CRI-O is an open-source container runtime that implements the Kubernetes Container Runtime Interface (CRI) to manage container images, pods, and containers on cluster nodes using OCI-compatible runtimes. It serves as a node-level container manager that handles image pulling, container lifecycle, and resource monitoring for Kubernetes clusters, running containers according to the Open Container Initiative specifications. The runtime distinguishes itself through live configuration reloading that applies changes to runtime definitions, registry mirrors, and TLS certificates without restarting th

Composer 是一个 PyTorch 分布式训练框架，旨在实现大规模模型在多节点 GPU 集群上的扩展。它兼具大语言模型训练器、分布式模型优化器和训练生命周期管理器的功能。 该项目作为深度学习正则化库脱颖而出，提供诸如 Sharpness Aware Minimization、MixUp 和 CutMix 等专业优化技术，以提升模型的泛化能力。它还通过序列长度预热、渐进式层冻结以及用于大规模模型恢复的分片状态检查点技术，优化了训练流程。 该框架涵盖了广泛的功能领域，包括分布式训练编排、混合精度硬件管理和云原生数据流。它还为 GPU 内存诊断、训练发散检测和吞吐量跟踪提供了丰富的监控与可观测性工具。 该项目包含一个命令行启动器，可自动执行跨节点的分布式多 GPU 训练任务。

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself

SparkInternals 是一份技术参考和架构指南，详细介绍了 Apache Spark 分布式计算引擎的内部设计和实现。它作为大数据引擎分析的研究资料，重点关注系统如何管理集群执行以及驱动节点（Driver）、执行器（Executor）和工作节点（Worker）之间的交互。 该项目详细分解了逻辑计划如何转换为物理执行阶段。它专门分析了数据 Shuffle 操作、内存管理以及分布式作业调度协调的机制。 该文档涵盖了广泛的分布式计算功能，包括查询执行规划、数据依赖管理和内存缓存策略。它还研究了任务分配、并行执行以及用于故障恢复和数据持久化的过程。

TinyBase 是一个响应式数据存储和内存关系数据库，专为客户端状态持久化而设计。它作为一个本地优先（local-first）的同步引擎，使用无冲突复制数据类型（CRDT）和逻辑时钟合并分布式状态，以确保确定性的数据收敛。 该项目具有一个模式验证库，可将来自 Zod、Yup 和 TypeBox 等工具的外部定义转换为类型安全的存储定义。它为实时协作编辑提供了基础设施，利用与 Automerge、Yjs 和 PartyKit 的同步来维护跨多个客户端和服务器的一致状态。 能力范围包括具有表和外键的关系数据建模、类似 SQL 的查询和索引，以及用于分组变更的原子事务。它支持广泛的持久化适配器，包括浏览器存储、SQLite 和 Cloudflare Durable Objects。该系统还提供双向状态绑定和用于与 React、SolidJS 和 Svelte 集成的声明式组件。

This project is a Git-based AI session tracker and context manager designed to record AI agent interactions, transcripts, and tool usage directly into Git repositories. It functions as a system for capturing and indexing the reasoning behind code changes, linking AI prompts and responses to specific code commits to preserve developer intent. The tool distinguishes itself by using Git as a primary storage layer for session metadata, utilizing shadow branches and checkpoints to track agent state without polluting the main commit log. It includes specialized capabilities for auditing AI contribu

RLinf is a distributed reinforcement learning orchestrator and embodied AI training framework. It provides the infrastructure to train vision-language-action models and robotic policies using a combination of reinforcement learning and supervised fine-tuning. The system is designed for scaling workloads across GPU clusters, managing the placement of actors, rollout workers, and environment components. It features a specialized robotics data collection pipeline for gathering teleoperated demonstrations and simulation trajectories into standardized replay buffers, alongside a hardware interface