Awesome GitHub RepositoriesDistributed GPU Computing

Megatron-LM is a distributed transformer training library and large language model training framework designed to scale models across thousands of GPUs. It functions as a GPU-optimized deep learning toolkit and a scaling engine for mixture-of-experts architectures, enabling the training of models with hundreds of billions of parameters. The project implements multi-dimensional model parallelism, combining tensor, pipeline, data, expert, and context-based workload distribution. It specifically optimizes mixture-of-experts architectures through integrated memory and communication improvements t

Horovod is a distributed deep learning framework designed to scale machine learning training across multiple GPUs and nodes. It functions as an orchestrator for multi-GPU scaling and a tool for distributed gradient averaging, allowing users to increase compute capacity without rewriting core model logic. The project provides a consistent communication interface that supports multi-framework model distribution across TensorFlow, PyTorch, Keras, and MXNet. It leverages an MPI distributed training library to synchronize gradients across processes using collective communication operations. The s

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

Petals is a decentralized framework and inference engine for running large language models across a peer-to-peer network. It enables the execution of models that exceed the memory of any single machine by splitting computations and model layers across a collaborative swarm of GPUs. The system functions as a collaborative compute network where participants share local GPU resources and host model weights. It supports distributed prompt-tuning to adapt massive models to specific tasks and allows for the establishment of private compute swarms to process sensitive data within restricted, trusted

cuDF is a GPU-accelerated dataframe library and data processing engine designed for manipulating and analyzing large tabular datasets. It provides a high-level API for executing filtering, joining, and aggregating operations directly on GPU hardware. The project integrates the Apache Arrow memory format to enable zero-copy data transfers and includes a just-in-time compiler for executing custom user-defined functions on the GPU. The library features specialized acceleration for existing workflows by redirecting standard Pandas dataframe calls and Polars query plans to a GPU backend. It also p

TensorTrade is a reinforcement learning trading framework designed for training and deploying autonomous agents that optimize financial market strategies. It provides an algorithmic trading simulation environment where agents can be tested against market data using simulated broker environments. The framework features a distributed training system using RLlib to optimize decision policies across large datasets. It includes a walk-forward validation tool that evaluates trading strategies through windowed performance analysis to prevent overfitting and measure real-world viability. The project

这是一个文本到图像 Transformer 的 PyTorch 实现。它是一个生成式 AI 模型，旨在利用 Transformer 网络将离散文本令牌映射到图像像素，从而从文本描述中创建视觉内容。 该系统利用离散 VAE 图像编码器将视觉数据压缩为令牌以供 Transformer 处理。它支持无分类器引导（classifier-free guidance）以在推理期间调整文本提示的影响，并包括根据与文本提示的相似度对生成图像进行排序的功能。 该架构结合了稀疏注意力机制和可逆残差网络，以优化计算复杂度和内存消耗。训练功能包括分布式 GPU 扩展，以及用于管理跨多个图形处理器的大规模工作负载以将图像与文本描述关联的框架。 该实现通过集成预训练的分词器或语言模型，提供对自定义文本分词的支持。

This is a PyTorch deep learning implementation for training transformer-based language models. It functions as a distributed GPU trainer and framework designed to optimize text prediction models for increased speed and sample efficiency. The project is distinguished by its use of the Newton-Schulz weight optimizer. This method applies an iterative process to maintain semi-orthogonal parameter updates and weight matrices, which improves sample efficiency and reduces memory overhead during the training process. The framework covers broad capabilities in distributed GPU computing, including dat

cuml is a GPU-accelerated machine learning library and framework that uses CUDA to accelerate tabular data preprocessing and model execution. It provides a suite of tools for training and deploying classification, regression, and clustering models on NVIDIA GPUs and GPU clusters. The library is designed for scalability, offering a distributed GPU machine learning environment that can spread computation and data across multiple hardware accelerators and nodes to handle datasets exceeding single-device memory. It mirrors standard estimator interfaces to allow the replacement of CPU-based models

TransformerLab 是一个 MLOps 编排平台和研究环境，专为大语言模型的训练、微调和评估而设计。它作为中央控制平面，用于管理机器学习任务，并协调跨混合云和本地提供商的分布式 GPU 计算。 该平台的特色在于代理驱动的模型优化，利用 AI 助手分析指标并自动提出和排队超参数实验。它提供了一个远程开发环境，允许用户直接在远程计算节点上启动交互式 Notebook、代码编辑器和 SSH 会话。 该系统涵盖了广泛的机器学习工作流功能，包括分布式任务协调、自动化超参数搜索和全面的实验跟踪。它具有用于版本化数据集和模型制品的集成注册表，以及用于模型性能评估和推理服务器部署的工具。 平台提供了命令行界面，用于平台控制、任务监控以及管理本地服务器实例的安装和更新。

StableSwarmUI 是一个用于 Stable Diffusion 图像生成的 Web 界面和后端编排器。它作为一个分布式 GPU 图像生成器和模块化 AI 图像流水线，提供了一个集中式控制器来管理图像生成请求。 该系统通过将生成任务拆分到多个图形处理器以提高批处理吞吐量的能力而脱颖而出。它利用后端无关的接口连接到本地服务器、远程服务器和云 API，并包含一个用于定义复杂图像处理操作的基于图的可视化工作流设计器。 该平台包括用于添加自定义功能的动态插件扩展系统，以及用于配置系统级依赖的自动化实用程序。它将模块化生成工具和快速编辑界面与跨分布式硬件路由工作负载的能力相结合。

NCCL 是一个高性能通信库和分布式 GPU 计算框架，专为在单节点或多节点系统中的多个 GPU 之间执行集合和点对点数据交换而设计。它充当 RDMA GPU 传输层和内存编排器，为分布式 GPU 训练和推理提供高带宽的数据和模型梯度同步。 该库的特色在于能够直接从 GPU 内核执行通信原语，将主机 CPU 从关键路径中移除。它利用拓扑感知路径选择来优化数据移动，并采用包括 InfiniBand 和 NVLink 在内的基于 RDMA 的网络传输，以实现设备跨不同物理节点之间的零拷贝内存访问。 该项目涵盖了广泛的集合通信模式，包括归约（Reductions）、广播（Broadcasts）、收集（Gathers）和全对全交换（All-to-all exchanges），以及点对点远程内存访问。它提供全面的通信器管理，用于初始化、分区和调整 GPU 组大小，以及用于注册缓冲区和协调共享设备内存的专用内存管理。 该系统包括一套用于健康跟踪、诊断日志记录和实时事件监控的监控与可观测性工具，以及用于机器学习框架、CUDA Graphs、MPI 和 Python 的集成接口。

Amazon DSSTNE 是一个机器学习工具包和稀疏张量网络库，专为具有稀疏输入和输出的深度学习模型而设计。它提供了一个模型并行训练框架和一个 GPU 加速的稀疏引擎，以支持内存密集型网络。 该框架专门为推荐系统训练和大规模稀疏学习而设计。它实现了将大型权重矩阵和嵌入表分布在多个 GPU 设备上，以处理超过单个处理器内存容量的模型。 该项目涵盖了广泛的能力，包括分布式 GPU 计算、稀疏数据集处理以及可扩展稀疏张量网络的构建。这些实用程序允许在 GPU 集群上执行高性能机器学习操作和模型扩展。

SLIME is a distributed reinforcement learning framework for large language model post-training that bridges Megatron training with SGLang inference servers. It orchestrates scalable RL loops across GPU clusters, decoupling training and inference into independent processes that communicate over HTTP and NCCL for independent scaling and fault tolerance. The system supports multi-agent reinforcement learning workflows with parallel agent instances, customizable rollout strategies, and personalized agent serving that improves models from prior conversations without disrupting API serving. The fra

CML 是一个用于训练和评估机器学习模型的管道自动化工具，作为机器学习的 CI/CD 系统运行。它作为一个云计算编排器和基于 Git 的工作流管理器，通过分支管理、自动提交和集成报告来自动化模型训练周期。 该项目通过配置临时云实例或 Kubernetes 节点来提供计算密集型任务所需的专用硬件，从而脱颖而出。它还管理远程计算运行器，允许连接自托管 GPU 集群或本地机器来执行容器化机器学习工作流。 该系统涵盖了广泛的功能，包括 ML 实验跟踪（性能指标和可视化直接发布到版本控制 Pull Request 中）。它处理从初始数据导入和版本控制到生成格式化工作流报告和外部可视化链接的 ML 管道自动化。 该工具通过基于 SSH 的远程调试和恢复中断作业的能力，为基础设施管理提供了额外的实用性。

Acme 是一个强化学习框架和执行引擎，旨在开发和基准测试学习算法。它提供了一个模块化组件库和参考实现，用于构建智能体并建立性能基准。 该系统支持将智能体架构从单流执行扩展到大规模分布式环境。这使得从初步原型设计到用于训练和评估的分布式执行的过渡变得更加顺畅。 该框架涵盖了强化学习开发和智能体架构原型设计，提供了将新模型与标准参考智能体进行基准测试所需的构建模块。

Discoart is a diffusion model orchestration framework and distributed GPU generation engine designed to automate and scale image generation workflows across hardware clusters. It functions as a generative AI model API, providing HTTP and gRPC endpoints to trigger and retrieve images from diffusion models as a network service. The system distinguishes itself through a comprehensive task management layer that includes timeline-based prompt and parameter scheduling. It manages the generative art lifecycle by supporting state-based session serialization for recovery, YAML-based configuration mana

IsaacGymEnvs is a GPU-accelerated physics sandbox and robotics policy training suite designed for reinforcement learning. It serves as a vectorized robotic simulator that runs thousands of parallel environments on GPUs to accelerate the training of neural networks. The project provides a sim-to-real transfer framework that utilizes domain randomization and physics variations to ensure policies trained in simulation are robust enough for deployment on real hardware. It distinguishes itself through a high-performance architecture that uses tensor-based state management to handle observations an

RLinf is a distributed reinforcement learning orchestrator and embodied AI training framework. It provides the infrastructure to train vision-language-action models and robotic policies using a combination of reinforcement learning and supervised fine-tuning. The system is designed for scaling workloads across GPU clusters, managing the placement of actors, rollout workers, and environment components. It features a specialized robotics data collection pipeline for gathering teleoperated demonstrations and simulation trajectories into standardized replay buffers, alongside a hardware interface