Awesome GitHub RepositoriesLarge-Scale Training Frameworks

Keras is a high-level deep learning API used to design, build, and train neural networks for tasks such as computer vision, natural language processing, and time series forecasting. It provides a framework for defining model architectures and optimizing weights through a structured interface. The project is defined by a backend-agnostic design that allows the same model code to run across different compute engines. This multi-backend execution enables users to swap underlying engines to optimize for specific hardware or performance requirements. The system supports distributed model training

DeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization

This repository serves as a comprehensive research platform and toolkit for advancing machine learning, quantum computing, and large-scale scientific data analysis. It provides foundational frameworks for developing complex algorithmic systems, offering the necessary infrastructure for distributed training, computational graph execution, and high-performance model development. The project distinguishes itself by integrating specialized research domains with robust, privacy-preserving methodologies. It supports diverse scientific discovery through tools for quantum simulation, physics-informed

Caffe is a high-performance deep learning framework designed for training and deploying deep neural networks. It functions as a machine learning engine and a convolutional neural network library, providing a C++ backend to accelerate computations on both GPUs and CPUs. The system includes a specialized toolset for computer vision, enabling tasks such as object detection, semantic segmentation, and large-scale image retrieval. It supports the deployment of pre-trained models for image and scene recognition, as well as the ability to fine-tune neural network weights for specialized tasks. The

Lightning is a PyTorch training framework and distributed AI training orchestrator designed to decouple core research logic from the engineering boilerplate required for model training. It functions as a deep learning workflow manager that automates the process of pretraining and finetuning models across diverse compute environments. The project distinguishes itself by providing a hardware-agnostic training wrapper, allowing the same model code to execute on CPUs, GPUs, or TPUs without modification. It further manages the scaling of workloads from single devices to multi-node clusters and ser

This project is a low-dependency engine designed for training large language models using native C and CUDA. It provides a bare-metal environment for tensor computation, allowing for the execution of neural network operations directly on hardware accelerators without the overhead of high-level software abstractions. The framework distinguishes itself by implementing manual gradient backpropagation and custom hardware-specific kernels, providing granular control over memory mapping and computational precision. It supports distributed training across multiple graphics processors and compute nod

PyTorch Geometric is a library for building and training machine learning models on graph-structured data. It provides a framework for developing graph neural networks, including a specialized system for implementing node-to-node information exchange via customizable message passing, aggregation, and update functions. The library includes a sparse data processing toolkit that utilizes accelerated CPU and CUDA kernels to perform efficient reductions on large sparse datasets. It supports the creation of specialized architectures for structured data such as 3D meshes and point clouds. The proje

This project is a deep learning library designed for training neural networks on irregular data structures, including graphs, 3D meshes, and point clouds. It functions as an extension to the PyTorch framework, providing specialized layers and kernels that enable the processing of complex, non-Euclidean information. The library distinguishes itself through a geometric deep learning toolkit that manages the unique requirements of graph-based data. It utilizes sparse matrix-based message passing to aggregate information across nodes and employs dynamic computational graph construction to accommo

Paddle is a deep learning framework designed for building, training, and deploying neural networks. It provides a platform for constructing models using tensor-based computations and supports both dynamic and static execution graphs to facilitate research and production workflows. The platform functions as a distributed machine learning system, enabling the scaling of training workloads across multiple nodes and hardware clusters. It includes a comprehensive toolkit for model deployment and optimization, allowing users to convert external model formats, compress trained models for resource-co

Apache MXNet is a deep learning framework and distributed machine learning library designed for training and deploying neural networks across distributed systems, mobile devices, and hardware accelerators. It functions as a cross-platform runtime and a dynamic dataflow scheduler that optimizes neural network execution. The framework provides a multi-language API, enabling the development of machine learning models using Python, R, Julia, Scala, Go, and JavaScript. It supports high-performance model training and the scaling of workloads across multiple GPUs and machines. The system covers cap

This project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip

Agent Lightning is an optimization framework designed to refine the performance of individual AI agents within complex multi-agent systems. It provides a platform for improving decision-making and task execution by applying reinforcement learning, supervised fine-tuning, and automated prompt optimization. The framework distinguishes itself through its ability to isolate specific agents for targeted tuning, allowing developers to enhance individual behaviors while maintaining the stability of the broader system architecture. By utilizing a modular interface, it integrates with diverse agent fr

This project is a comprehensive toolkit designed for the full lifecycle management of large language and multimodal models. It functions as a unified orchestrator that handles the entire development process, ranging from dataset preparation and supervised fine-tuning to advanced reinforcement learning alignment and production-ready inference deployment. The platform distinguishes itself through a specialized reinforcement learning library that supports complex optimization algorithms, including group relative policy optimization and leave-one-out techniques, to improve model instruction-follo

This project is a curated directory of software, frameworks, and educational resources designed for building, scaling, and maintaining distributed data processing and storage architectures. It serves as a comprehensive index for the distributed computing ecosystem, helping users identify the appropriate tools for managing large-scale information systems. The repository functions as a central hub for data engineering, offering categorized access to technologies that support batch and stream processing, machine learning, and interactive querying. By organizing these resources, it assists in the

DGL is a Python library for building and training graph neural networks. It functions as a graph message passing framework and a geometric deep learning tool, enabling the development of models that analyze graph-structured data. The library is designed for large-scale graph processing, utilizing distributed training and neighbor sampling to handle datasets with billions of edges. It provides specialized support for heterogeneous graph modeling, allowing for the representation of complex real-world entities with multiple node and edge types. Its capabilities cover a wide range of graph tasks

Monolith is a distributed recommendation model framework and asynchronous training engine designed to build and train large-scale deep learning architectures. It functions as a distributed model trainer that processes massive datasets across multiple compute nodes using asynchronous update mechanisms. The system features a dedicated embedding table manager that creates unique, feature-isolated tables to prevent representation collisions. It also includes a real-time weight updater to capture immediate changes in user interest and data hotspots through continuous parameter synchronization. Th

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

Corenet is a deep learning training framework and computer vision model library designed for developing neural networks across vision, text, and audio modalities. It functions as a distributed training orchestrator for scaling workloads across multiple compute nodes and provides a multimodal data pipeline for processing image, text, and video data. The project includes a model conversion toolkit for transforming weights and architectures between different machine learning frameworks. It also provides tools for optimizing model performance on Apple Silicon and reducing response latency in gene

Composer es un framework de entrenamiento distribuido para PyTorch diseñado para escalar modelos a gran escala en clústeres de GPU multi-nodo. Funciona como un entrenador de modelos de lenguaje de gran tamaño (LLM), un optimizador de modelos distribuidos y un gestor del ciclo de vida de entrenamiento. El proyecto se diferencia como una biblioteca de regularización para deep learning, proporcionando técnicas de optimización especializadas como Sharpness Aware Minimization, MixUp y CutMix para mejorar la generalización del modelo. Además, distingue su flujo de entrenamiento mediante el uso de warmup de longitud de secuencia, congelación progresiva de capas y checkpointing de estado fragmentado (sharded-state) para la recuperación de modelos a gran escala. El framework cubre una amplia superficie de capacidades, incluyendo la orquestación de entrenamiento distribuido, la gestión de hardware de precisión mixta y el streaming de datos cloud-native. También proporciona herramientas extensas de monitoreo y observabilidad para diagnósticos de memoria de GPU, detección de divergencia en el entrenamiento y seguimiento del rendimiento (throughput). El proyecto incluye un lanzador de línea de comandos para automatizar la ejecución de trabajos de entrenamiento multi-GPU entre nodos.

Flashlight es una biblioteca de aprendizaje automático en C++ y un framework de aprendizaje profundo diseñado para construir y entrenar redes neuronales. Funciona como una biblioteca de manipulación de tensores y un motor de diferenciación automática que rastrea operaciones para calcular gradientes mediante retropropagación (backpropagation) para la optimización de modelos. El proyecto se distingue por su rol como framework de entrenamiento distribuido, utilizando sincronización de gradientes all-reduce y entornos distribuidos para escalar cargas de trabajo de aprendizaje automático a través de múltiples nodos y dispositivos. Cuenta con una interfaz de memoria agnóstica al backend y gestión basada en RAII para desacoplar las operaciones de tensores del hardware físico. El framework cubre una amplia superficie de capacidades, incluyendo la construcción de arquitecturas de redes neuronales con capas convolucionales, lineales y recurrentes. Proporciona utilidades extensas para álgebra de tensores, gestión y batching de datasets, serialización binaria versionada para estados de modelos y herramientas de monitorización para rastrear métricas de entrenamiento y uso de memoria.