Awesome GitHub RepositoriesDistributed Training Sharding

DeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en

This library provides a framework for parameter-efficient fine-tuning, enabling the adaptation of large pretrained models by training only a small subset of parameters. It functions as a distributed model training system and optimization toolkit, designed to reduce the computational and memory requirements typically associated with full model fine-tuning. The project distinguishes itself through a suite of methods for modular adapter composition, including low-rank matrix decomposition and activation-based scaling. It supports the integration of multiple task-specific adapter modules, allowin

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

Accelerate is a PyTorch distributed training library that abstracts the boilerplate required to run models across multiple GPUs, TPUs, and CPUs. It functions as a deep learning model scaler and distributed hardware orchestrator, allowing the same training script to run on different hardware backends without modifying the core logic. The project provides a distributed training command line interface for configuring compute environments and launching jobs across single or multi-node clusters. It includes a mixed precision training framework to implement FP16 and BF16 precision, reducing memory

OpenRLHF is a training framework and alignment library designed for reinforcement learning from human feedback across distributed GPU clusters. It provides tools for aligning large language models and multimodal vision-language models using algorithms such as PPO, GRPO, and DPO. The framework distinguishes itself through a distributed inference engine that overlaps sample rollout with training to increase throughput. It supports scaling to models exceeding 70 billion parameters via parameter sharding and handles long-context sequences through ring-attention sequence parallelism. The project

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

bitsandbytes is a deep learning quantization tool and library designed to reduce the memory footprint of large language models. It serves as a GPU memory optimizer and quantization framework, compressing model weights and features to 8-bit and 4-bit precision to enable inference and training on hardware with limited memory. The project provides a framework for low-rank adaptation, allowing the fine-tuning of quantized models by combining 4-bit weights with small trainable matrices. It further distinguishes itself through memory paging, which moves optimizer states between CPU and GPU memory t

该项目是一个基于 JAX 的 Transformer 框架和大语言模型训练器，专为在 TPU 硬件加速器上构建和训练分布式模型而设计。它提供了一个通过在设备网格上拆分权重和计算来减少内存开销并提高处理速度的系统，用于预训练和微调自回归模型。 该框架包含一个 TPU 计算编排器，用于资源配置和自动化远程分布式节点上的依赖安装。它还具有一个模型权重转换器，能够在不同的硬件配置和数值精度之间转换和重新分片检查点。 该项目涵盖了更广泛的功能，包括用于云存储的分片检查点管理、具有状态恢复的流式数据加载，以及用于模型推理的基于核的文本生成。它进一步支持针对 TPU 和 GPU 集群的 XLA 编译硬件加速，并提供针对标准化语言任务进行性能基准测试的工具。

This repository is the official documentation for TensorFlow, a machine learning framework. It provides comprehensive guides, tutorials, and API references for building, training, and deploying machine learning models. The documentation covers the full lifecycle of machine learning projects, from constructing data pipelines and building neural networks with high-level APIs to customizing training loops and deploying trained models in production, on edge devices, or in browsers. The documentation includes step-by-step tutorials for a range of tasks, including reinforcement learning, ranking mo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

llm-foundry 是一个大型语言模型训练框架，提供了一个用于基础模型预训练和监督微调的系统。它包括一个用于跨多个节点和 GPU 扩展工作负载的分布式训练器、一个用于从云存储加载数据的数据集流式传输管道，以及参数高效的微调实现。 该框架通过使用参数分片和高吞吐量数据流来保持大规模训练期间的稳定性，从而脱颖而出。它结合了低秩自适应（LoRA）以降低计算成本，并使用 8 位浮点精度来提高兼容硬件上的计算速度。 该代码库涵盖了广泛的功能，包括将原始数据转换为压缩格式的数据集工程、通过评估套件进行的模型性能基准测试，以及将模型权重导出为标准化行业格式的能力。它还支持通过装饰器进行自定义组件注册，并提供对位置嵌入方法的控制。

该项目是 PyTorch 深度学习框架技术指南和 API 参考的中文翻译版。它作为一个本地化的知识库和参考资料，使非英语母语者能够访问深度学习文档。 该文档涵盖了 PyTorch 的全面能力，包括神经网络模型开发、自动微分以及后端内核的实现。它提供了关于分布式训练策略、通过 ONNX 和 C++ 等格式进行模型部署以及各种模型优化和量化技术的详细指导。 该项目利用社区驱动的翻译管道和分布式贡献模型来维护版本同步的内容。技术资料使用 Markdown 组织，并通过静态网站生成渲染为可导航的网站。

Fengshenbang-LM 是一个中文语言模型生态系统和预训练框架，专为开发和微调十亿参数级大语言模型而设计。它作为一个自然语言处理流水线和跨模态 AI 平台，能够生成跨模态内容，包括文生图和蛋白质结构预测。 该项目提供了一个特定领域模型适配器，用于将预训练模型应用于医疗、金融和法律等专业行业。它利用分布式配置系统和数据分片来管理跨多个计算节点的大规模模型训练。 该框架涵盖了广泛的功能，包括中文自然语言处理、自动文本转换和多模态内容生成。它还支持翻译、编程和文本分类等通用语言任务。 下游预测和微调工作流通过命令行界面进行管理和执行。

OpenManus-RL is a reinforcement learning framework and distributed training pipeline designed to train large language models as agents. It serves as an agentic reasoning optimizer and reward model trainer, providing the infrastructure to improve model decision-making through reward-based policy optimization. The project distinguishes itself through a distributed architecture that supports parameter sharding across multiple compute nodes and a coordinated rollout system for collecting interaction trajectories. It incorporates advanced reasoning strategies, such as Tree-of-Thoughts and Monte Ca

Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt