Awesome GitHub RepositoriesCommunication Kernels

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat

NCCL 是一个高性能通信库和分布式 GPU 计算框架，专为在单节点或多节点系统中的多个 GPU 之间执行集合和点对点数据交换而设计。它充当 RDMA GPU 传输层和内存编排器，为分布式 GPU 训练和推理提供高带宽的数据和模型梯度同步。 该库的特色在于能够直接从 GPU 内核执行通信原语，将主机 CPU 从关键路径中移除。它利用拓扑感知路径选择来优化数据移动，并采用包括 InfiniBand 和 NVLink 在内的基于 RDMA 的网络传输，以实现设备跨不同物理节点之间的零拷贝内存访问。 该项目涵盖了广泛的集合通信模式，包括归约（Reductions）、广播（Broadcasts）、收集（Gathers）和全对全交换（All-to-all exchanges），以及点对点远程内存访问。它提供全面的通信器管理，用于初始化、分区和调整 GPU 组大小，以及用于注册缓冲区和协调共享设备内存的专用内存管理。 该系统包括一套用于健康跟踪、诊断日志记录和实时事件监控的监控与可观测性工具，以及用于机器学习框架、CUDA Graphs、MPI 和 Python 的集成接口。

AITemplate 是一个提前（ahead-of-time）深度学习编译器，将 PyTorch 神经网络转换为独立的 C++ 源代码。它作为一个 PyTorch 转 C++ 编译器和 GPU 内核融合引擎，生成自包含的可执行二进制文件，无需 Python 解释器或深度学习框架运行时即可运行推理。 该项目专门为 NVIDIA TensorCores 和 AMD MatrixCores 生成优化的 CUDA 和 HIP C++ 代码。它专注于通过将多个神经网络算子组合成单个 GPU 内核的系统，最大限度地提高半精度浮点运算的吞吐量，从而最小化内存开销和延迟。 该工具集涵盖了 GPU 推理加速和高性能计算，提供了自定义 GPU 算子开发以及将图节点映射到硬件特定模板的能力。它包括用于基准测试推理性能和可视化模型优化的实用支持。

FastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc