Open-source alternatives to Nccl

BytePS is a distributed deep neural network training framework and communication library designed to scale model training across multiple GPUs and compute nodes. It functions as a GPU cluster orchestrator and RDMA network optimizer, providing the necessary primitives to synchronize gradients and data across a server cluster. The project distinguishes itself through high-performance network optimizations, utilizing remote direct memory access and page-aligned memory to reduce latency. It employs topology-aware communication tuning and CPU core affinity management to maximize hardware throughpu

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

Boost is a collection of portable, high-performance source libraries that extend the C++ standard library. It provides a wide range of reusable components, data structures, and algorithms designed to add capabilities to the base language across different platforms. The project is distinguished by its extensive focus on compile-time template metaprogramming and generic programming. It implements advanced architectural patterns such as policy-based design, concept-based type validation, and the use of SFINAE for conditional template resolution to minimize runtime overhead. The library covers a

Mooncake is a disaggregated large language model serving platform and distributed key-value store designed for high-performance inference infrastructure. It functions as a GPU memory orchestrator and KV cache management system that pools and transfers key-value caches across clusters to accelerate inference. The system distinguishes itself by separating the prefill and decode phases of inference into distinct hardware clusters to optimize resource utilization. It utilizes a high-performance RDMA distributed cache with zero-copy transfers to move data between compute nodes, bypassing the CPU t

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Amazon DSSTNE is a machine learning toolkit and sparse tensor network library designed for deep learning models with sparse inputs and outputs. It provides a model-parallel training framework and a GPU-accelerated sparse engine to support memory-intensive networks. The framework is specifically designed for recommendation system training and large-scale sparse learning. It enables the distribution of large weight matrices and embedding tables across multiple GPU devices to handle models that exceed the memory capacity of a single processor. The project covers a broad range of capabilities in

This is a PyTorch deep learning implementation for training transformer-based language models. It functions as a distributed GPU trainer and framework designed to optimize text prediction models for increased speed and sample efficiency. The project is distinguished by its use of the Newton-Schulz weight optimizer. This method applies an iterative process to maintain semi-orthogonal parameter updates and weight matrices, which improves sample efficiency and reduces memory overhead during the training process. The framework covers broad capabilities in distributed GPU computing, including dat

Horovod is a distributed deep learning framework designed to scale machine learning training across multiple GPUs and nodes. It functions as an orchestrator for multi-GPU scaling and a tool for distributed gradient averaging, allowing users to increase compute capacity without rewriting core model logic. The project provides a consistent communication interface that supports multi-framework model distribution across TensorFlow, PyTorch, Keras, and MXNet. It leverages an MPI distributed training library to synchronize gradients across processes using collective communication operations. The s

Megatron-LM is a distributed transformer training library and large language model training framework designed to scale models across thousands of GPUs. It functions as a GPU-optimized deep learning toolkit and a scaling engine for mixture-of-experts architectures, enabling the training of models with hundreds of billions of parameters. The project implements multi-dimensional model parallelism, combining tensor, pipeline, data, expert, and context-based workload distribution. It specifically optimizes mixture-of-experts architectures through integrated memory and communication improvements t

Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and

Ignite is a high-level training framework for PyTorch neural networks that serves as a training engine and deep learning lifecycle manager. It provides a structured system for organizing and automating training and evaluation loops, managing data iterators and triggering event handlers at specific milestones during the model training process. The project distinguishes itself through a comprehensive suite of tools for distributed training and model evaluation. It includes utilities for synchronizing gradients and coordinating collective communication across multiple GPUs or nodes, as well as a

IREE is an MLIR-based compiler toolchain and runtime designed to translate machine learning models from various frameworks into optimized binaries for execution across diverse hardware targets. It provides a unified pipeline to ingest models from PyTorch, TensorFlow, JAX, and ONNX, lowering them into a common intermediate representation for deployment on CPUs, GPUs, and bare-metal embedded systems. The project distinguishes itself through a bytecode virtual machine and a hardware abstraction layer that decouple high-level model logic from specific hardware instruction sets. It supports sophis

This project is a PyTorch implementation of a text-to-image transformer. It is a generative AI model designed to map discrete text tokens to image pixels using a transformer network to create visual content from textual descriptions. The system utilizes a discrete VAE image encoder to compress visual data into tokens for transformer processing. It supports classifier-free guidance to adjust the influence of text prompts during inference and includes capabilities for ranking generated images based on their similarity to text prompts. The architecture incorporates sparse attention mechanisms a

This project provides a comprehensive technical guide and framework for engineering large-scale machine learning systems. It covers the full lifecycle of model development, focusing on the infrastructure and computational principles required to build, train, and serve generative AI models across distributed GPU clusters. The repository distinguishes itself by offering deep-dive tutorials and implementation strategies for complex system challenges. It emphasizes high-performance architectural primitives, such as collective communication orchestration, distributed tensor sharding, and static gr

zero_nlp is a distributed framework for training and fine-tuning large language models and multimodal architectures. It provides a specialized toolkit for distributed model parallelism, allowing neural network layers and weights to be partitioned across multiple GPU devices to train models that exceed the memory capacity of a single processor. The project distinguishes itself through a combination of high-throughput data pipelines and parameter-efficient tuning. It utilizes multi-threading and memory mapping to preprocess and stream datasets exceeding 100GB and implements memory-saving adapta

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

This project is an MLOps architectural guide and framework for designing and deploying deep learning systems into production environments. It provides a structured approach to model inference deployment, ML pipeline orchestration, and the creation of production-level machine learning architectures. The project distinguishes itself through a focus on distributed deep learning and edge AI optimization. It covers methodologies for parallelizing model training across multiple GPUs to handle large datasets and applies techniques like quantization and distillation to reduce model size for embedded

This project is a self-supervised contrastive learning framework designed to train deep learning models to learn visual representations from images without using human-provided labels. It provides a system for developing pretrained visual representation models that can be adapted for downstream computer vision tasks. The framework includes tools for semi-supervised image classification, which combines large unlabeled datasets with small labeled sets to improve accuracy. It also features a linear probe evaluation tool to assess the quality of learned image features by training a simple linear

This project is a diffusion model framework for training and sampling from denoising probabilistic models to generate images from noise. It functions as a generative image model that creates visual content by iteratively refining random noise into coherent images. The system includes a distributed GPU trainer designed to scale complex neural network architectures across multiple graphics processing units. It also provides an image dataset preprocessor to prepare, scale, and standardize raw image collections for training. The framework covers model training and image generation, utilizing noi

FedML is a distributed machine learning training library, federated learning framework, and GPU workload orchestrator. It provides the core system components necessary to execute large-scale model training and fine-tuning across multi-cloud, on-premise, and decentralized GPU clusters, while offering a dedicated engine for scalable model serving and an MLOps pipeline manager for end-to-end lifecycle management. The platform distinguishes itself by enabling privacy-preserving federated learning across decentralized edge devices and organizational silos, keeping raw data on local hardware. It al

This project is a PyTorch implementation of StyleGAN2, providing a library and research framework for training style-based generative adversarial networks. It serves as a toolkit for high-resolution image synthesis, utilizing competitive minimax optimization to create realistic synthetic visual content. The framework incorporates specialized architectural components such as style-based latent mapping, multi-scale feature modulation, and self-attention layers to improve structural coherence. It distinguishes itself with advanced training stability techniques, including exponential moving avera

FastVideo is a comprehensive system for accelerated video generation, serving as a video generation inference engine, a video diffusion training framework, and a modular pipeline orchestrator. It provides a distributed transformer optimizer and a distillation toolkit designed to reduce denoising steps and model complexity to increase frame rates. The project distinguishes itself through specialized acceleration techniques, including joint distillation and sparse attention training. It implements low-step video generation and weight quantization to FP8 or FP4 precision to increase throughput a

h2o-llmstudio is a language model training framework that provides a no-code graphical interface for fine-tuning large language models on custom datasets. It functions as a specialized tool for managing the training lifecycle, from configuring hyperparameters to monitoring performance metrics. The project distinguishes itself through a multi-GPU training orchestrator that distributes workloads via data parallel processing and a low-rank adaptation tool for memory-efficient fine-tuning. It also includes a model evaluation dashboard featuring an interactive chat interface to verify conversation

Gluon-CV is an MXNet computer vision library that provides a comprehensive collection of pre-implemented vision architectures and training pipelines. It serves as a deep learning research toolkit and a model zoo containing state-of-the-art pre-trained weights for image and video analysis. The project includes a specialized human pose estimation library and a model compression toolkit. These tools allow for the pruning and quantization of deep learning models to increase inference speed and facilitate deployment on constrained edge hardware. The library covers a broad range of vision capabili

Super-Gradients is a PyTorch computer vision framework and training library designed for the full lifecycle of vision models. It functions as a deep learning model optimizer and a deployment toolkit for training and fine-tuning models across image classification, object detection, semantic segmentation, and pose estimation tasks. The project provides specific tools for model optimization, including teacher-student knowledge distillation and numerical precision compression to reduce memory and computational requirements. It also includes the implementation of the Yolo-NAS architecture for high