Awesome GitHub RepositoriesDistributed Training Sharding

DeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization

Fairseq is a PyTorch toolkit for sequence-to-sequence modeling, specializing in neural machine translation, automatic speech recognition, and large-scale language model training. It provides a framework for processing and aligning diverse data sources, including text, audio, and video, to support tasks such as speech-to-text conversion and multimodal sequence learning. The project is distinguished by its distributed training capabilities, which utilize parameter sharding, mixed-precision training, and CPU offloading to handle models that exceed single-device memory. It also includes specializ

This project is a machine learning array framework and tensor computation library designed for high-performance numerical computing. It provides a comprehensive suite of tools for constructing and training neural networks, featuring an automatic differentiation engine that facilitates gradient-based optimization and complex mathematical modeling. The library distinguishes itself through a unified memory architecture that allows data to be shared across CPU and GPU devices without explicit copies, significantly reducing data movement overhead. Its execution model relies on a lazy evaluation en

This library provides a framework for parameter-efficient fine-tuning, enabling the adaptation of large pretrained models by training only a small subset of parameters. It functions as a distributed model training system and optimization toolkit, designed to reduce the computational and memory requirements typically associated with full model fine-tuning. The project distinguishes itself through a suite of methods for modular adapter composition, including low-rank matrix decomposition and activation-based scaling. It supports the integration of multiple task-specific adapter modules, allowin

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

Accelerate is a PyTorch distributed training library that abstracts the boilerplate required to run models across multiple GPUs, TPUs, and CPUs. It functions as a deep learning model scaler and distributed hardware orchestrator, allowing the same training script to run on different hardware backends without modifying the core logic. The project provides a distributed training command line interface for configuring compute environments and launching jobs across single or multi-node clusters. It includes a mixed precision training framework to implement FP16 and BF16 precision, reducing memory

OpenRLHF is a training framework and alignment library designed for reinforcement learning from human feedback across distributed GPU clusters. It provides tools for aligning large language models and multimodal vision-language models using algorithms such as PPO, GRPO, and DPO. The framework distinguishes itself through a distributed inference engine that overlaps sample rollout with training to increase throughput. It supports scaling to models exceeding 70 billion parameters via parameter sharding and handles long-context sequences through ring-attention sequence parallelism. The project

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

Sana is a framework for high-resolution image and video synthesis based on a linear diffusion transformer. It provides a toolkit for the training, fine-tuning, and execution of text-to-image and text-to-video models, as well as a video generative world model capable of simulating physical environments with precise spatial control. The project is distinguished by its use of linear complexity layers to handle high resolutions and its support for long-form, minute-length video generation in real time. It implements a two-stage inference paradigm that separates structural generation from visual t

bitsandbytes is a deep learning quantization tool and library designed to reduce the memory footprint of large language models. It serves as a GPU memory optimizer and quantization framework, compressing model weights and features to 8-bit and 4-bit precision to enable inference and training on hardware with limited memory. The project provides a framework for low-rank adaptation, allowing the fine-tuning of quantized models by combining 4-bit weights with small trainable matrices. It further distinguishes itself through memory paging, which moves optimizer states between CPU and GPU memory t

Ce projet est un framework de transformer basé sur JAX et un entraîneur de modèles de langage large conçu pour construire et entraîner des modèles distribués sur des accélérateurs matériels TPU. Il fournit un système pour le pré-entraînement et le fine-tuning de modèles autorégressifs en divisant les poids et les calculs sur un maillage de périphériques pour réduire la surcharge mémoire et augmenter la vitesse de traitement. Le framework inclut un orchestrateur de calcul TPU pour provisionner les ressources et automatiser l'installation des dépendances sur des nœuds distribués distants. Il dispose également d'un convertisseur de poids de modèle capable de transformer et de re-sharder les checkpoints entre différentes configurations matérielles et précisions numériques. Le projet couvre des capacités plus larges, notamment la gestion de checkpoints shardés pour le stockage cloud, le chargement de données par flux avec restauration d'état et la génération de texte basée sur le noyau pour l'inférence. Il prend en charge l'accélération matérielle compilée XLA pour les clusters TPU et GPU et fournit des outils de benchmarking de performance sur des tâches linguistiques standardisées.

This repository is the official documentation for TensorFlow, a machine learning framework. It provides comprehensive guides, tutorials, and API references for building, training, and deploying machine learning models. The documentation covers the full lifecycle of machine learning projects, from constructing data pipelines and building neural networks with high-level APIs to customizing training loops and deploying trained models in production, on edge devices, or in browsers. The documentation includes step-by-step tutorials for a range of tasks, including reinforcement learning, ranking mo

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

MMF is a modular framework for building, training, and evaluating vision-and-language models. It provides a configuration-driven experiment system where model, dataset, and training parameters are defined through composable YAML files, alongside a curated model zoo of pretrained checkpoints for state-of-the-art multimodal architectures. The framework includes a multimodal dataset loader that downloads, processes, and batches vision-and-language data, and a vision-language model trainer supporting distributed training, mixed precision, and checkpoint-based resumption. The framework distinguish

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

llm-foundry est un framework d'entraînement pour grands modèles de langage, fournissant un système pour le pré-entraînement de modèles de fondation et le fine-tuning supervisé. Il inclut un entraîneur distribué pour mettre à l'échelle les charges de travail sur plusieurs nœuds et GPU, un pipeline de streaming de données pour charger les données depuis le stockage cloud, et une implémentation de fine-tuning efficace en paramètres. Le framework se distingue par son utilisation du sharding de paramètres et du streaming de données à haut débit pour maintenir la stabilité lors d'entraînements à grande échelle. Il intègre l'adaptation de bas rang (LoRA) pour réduire les coûts de calcul et utilise une précision en virgule flottante 8 bits pour augmenter la vitesse de calcul sur le matériel compatible. La base de code couvre un large éventail de capacités, incluant l'ingénierie de jeux de données pour transformer des données brutes en formats compressés, le benchmarking des performances des modèles via une suite d'évaluation, et la capacité d'exporter les poids des modèles dans des formats industriels standardisés. Il prend également en charge l'enregistrement de composants personnalisés via des décorateurs et offre un contrôle sur les méthodes d'embedding positionnel.

This project is a Chinese language translation of the technical guides and API references for the PyTorch deep learning framework. It serves as a localized knowledge base and reference material to make deep learning documentation accessible to non-English speakers. The documentation covers a comprehensive range of PyTorch capabilities, including neural network model development, automatic differentiation, and the implementation of backend kernels. It provides detailed guidance on distributed training strategies, model deployment through formats like ONNX and C++, and various model optimizatio

Fengshenbang-LM est un écosystème de modèles de langage chinois et un framework de pré-entraînement conçu pour le développement et le réglage fin de grands modèles de langage à plusieurs milliards de paramètres. Il sert de pipeline de traitement du langage naturel et de plateforme d'IA cross-modale capable de générer du contenu à travers différentes modalités, y compris la génération texte-vers-image et la prédiction de structure protéique. Le projet fournit un adaptateur de modèle spécifique au domaine pour appliquer des modèles pré-entraînés à des industries spécialisées telles que la santé, la finance et le droit. Il utilise un système de configuration distribué et le sharding de données pour gérer l'entraînement de modèles à grande échelle sur plusieurs nœuds de calcul. Le framework couvre un large éventail de capacités, y compris le traitement du langage naturel chinois, la transformation automatique de texte et la génération de contenu multimodal. Il prend en charge en outre des tâches linguistiques générales telles que la traduction, la programmation et la classification de texte. Les flux de travail de prédiction et de réglage fin en aval sont gérés et exécutés via une interface en ligne de commande.

OpenManus-RL is a reinforcement learning framework and distributed training pipeline designed to train large language models as agents. It serves as an agentic reasoning optimizer and reward model trainer, providing the infrastructure to improve model decision-making through reward-based policy optimization. The project distinguishes itself through a distributed architecture that supports parameter sharding across multiple compute nodes and a coordinated rollout system for collecting interaction trajectories. It incorporates advanced reasoning strategies, such as Tree-of-Thoughts and Monte Ca

Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt