Awesome GitHub RepositoriesDistributed Training Orchestration

Keras is a high-level deep learning API used to design, build, and train neural networks for tasks such as computer vision, natural language processing, and time series forecasting. It provides a framework for defining model architectures and optimizing weights through a structured interface. The project is defined by a backend-agnostic design that allows the same model code to run across different compute engines. This multi-backend execution enables users to swap underlying engines to optimize for specific hardware or performance requirements. The system supports distributed model training

PyTorch Lightning is a high-level deep learning framework for PyTorch that automates training loops and removes repetitive engineering boilerplate. It functions as a structured pipeline for managing machine learning experiments, providing a distributed training orchestrator and tools for mixed-precision training. The framework decouples scientific model architecture from the engineering required for infrastructure and scaling. This separation allows the same model code to execute across CPUs, GPUs, or TPUs through a hardware-agnostic execution engine and a centralized trainer that manages the

PyTorch Lightning is a deep learning research framework that provides a structured environment for organizing machine learning code. It functions as a unified trainer orchestrator, centralizing the execution flow by managing the interaction between hardware resources, data loaders, and model components. By decoupling model architecture from training logic, the framework enables researchers to maintain clean, modular codebases that remain portable across different environments. The framework distinguishes itself through a hardware-agnostic abstraction layer that scales deep learning workloads

This repository serves as a comprehensive collection of reference implementations for the PyTorch machine learning library. It provides practical examples for building, training, and deploying deep learning models, functioning as a toolkit for developers to explore neural network architectures and training workflows. The project distinguishes itself by offering concrete demonstrations of complex machine learning operations, ranging from computer vision tasks like object detection and depth estimation to the training of large-scale transformer models. These examples illustrate how to implement

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

This project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip

This project is a comprehensive engineering framework and technical reference for managing, scaling, and optimizing distributed machine learning infrastructure. It provides a suite of methodologies and diagnostic tools designed to support large-scale model training and inference on high-performance computing clusters. The project distinguishes itself through a specialized diagnostic toolkit and infrastructure optimization suite that addresses the complexities of multi-node environments. It enables precise control over cluster resources, including hardware maintenance, network topology configu

NeMo is a comprehensive framework designed for the development, training, and deployment of large-scale conversational and generative artificial intelligence models. It provides an integrated platform for building multimodal systems, encompassing speech processing, language modeling, and reinforcement learning alignment. The framework is built to handle the entire lifecycle of AI development, from data curation and model pretraining to production-ready service deployment. The platform distinguishes itself through advanced distributed training capabilities, including tensor and pipeline parall

AISystem is a comprehensive AI full-stack infrastructure project covering the entire pipeline from AI chip architecture to high-level training frameworks. It encompasses the development of AI compiler frameworks, inference engines, and distributed training orchestrators designed to coordinate workloads across a heterogeneous compute stack of CPUs, GPUs, and NPUs. The project focuses on the deep integration of software and hardware, employing software-hardware co-design to align tensor layouts with physical memory structures. It provides specialized capabilities for accelerating Transformer mo

Baselines is a comprehensive suite of frameworks for reinforcement learning algorithm implementation, imitation learning, and training orchestration. It provides a library of standardized learning algorithms used to benchmark and replicate research results, alongside a deep learning policy framework for constructing neural network architectures such as multi-layer perceptrons, convolutional networks, and long short-term memory networks. The project includes a specialized imitation learning toolkit that enables agents to mimic expert behavior through behavior cloning and generative adversarial

Agent Lightning is an optimization framework designed to refine the performance of individual AI agents within complex multi-agent systems. It provides a platform for improving decision-making and task execution by applying reinforcement learning, supervised fine-tuning, and automated prompt optimization. The framework distinguishes itself through its ability to isolate specific agents for targeted tuning, allowing developers to enhance individual behaviors while maintaining the stability of the broader system architecture. By utilizing a modular interface, it integrates with diverse agent fr

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi

NNI is an AutoML toolkit designed to automate machine learning lifecycles. It functions as a hyperparameter optimization framework, a neural architecture search tool, and a model compression suite. The project provides a distributed training orchestrator to manage machine learning workloads across local machines, remote servers, and cloud platforms. It enables the discovery of efficient model structures through reinforcement learning and one-shot optimization methods, while utilizing Bayesian and evolutionary algorithms to automate hyperparameter tuning. Additional capabilities include tools

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

Ludwig is a declarative machine learning framework designed for training neural networks and large language models using configuration files instead of manual coding. It functions as a multimodal model builder and a low-code tool for supervised fine-tuning, allowing users to build models that process mixed inputs of text, images, audio, and tabular data. The project distinguishes itself through an automated hyperparameter optimizer and a system for large language model fine-tuning using parameter-efficient adapters. It features a multimodal data pipeline and the ability to automatically gener

s1 is a reasoning training framework and GPU cluster orchestrator designed to build and refine large language models. It provides a system for executing supervised fine-tuning on distributed hardware, utilizing gradient checkpointing and hardware optimization to improve model reasoning. The project features a synthetic data generator and dataset builder that produce high-quality training sets. This workflow collects questions, generates model reasoning traces, and applies automated grading loops to filter for correct answers. The framework includes an evaluation suite to compute accuracy and

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

Composer est un framework d'entraînement distribué PyTorch conçu pour mettre à l'échelle des modèles de grande taille sur des clusters GPU multi-nœuds. Il fonctionne comme un entraîneur de grands modèles de langage, un optimiseur de modèle distribué et un gestionnaire de cycle de vie d'entraînement. Le projet se différencie en tant que bibliothèque de régularisation pour le deep learning, fournissant des techniques d'optimisation spécialisées telles que Sharpness Aware Minimization, MixUp et CutMix pour améliorer la généralisation des modèles. Il distingue davantage son flux d'entraînement par l'utilisation du warmup de longueur de séquence, du gel progressif des couches et du checkpointing d'état fragmenté pour la récupération de modèles à grande échelle. Le framework couvre une large surface de capacités, incluant l'orchestration de l'entraînement distribué, la gestion du matériel en précision mixte et le streaming de données cloud-native. Il fournit également des outils étendus de surveillance et d'observabilité pour les diagnostics de mémoire GPU, la détection de divergence d'entraînement et le suivi du débit. Le projet inclut un lanceur en ligne de commande pour automatiser l'exécution de tâches d'entraînement multi-GPU sur plusieurs nœuds.

ZenML is an extensible machine learning orchestration framework designed to manage the end-to-end lifecycle of data pipelines and AI agent workflows. It functions as a durable orchestrator that executes machine learning tasks as directed acyclic graphs, ensuring that every step is containerized for consistent performance across local, cloud, and hybrid infrastructure. By decoupling pipeline code from underlying compute and storage backends, the platform allows developers to define infrastructure-agnostic stacks that remain portable across diverse environments. The project distinguishes itself