Awesome GitHub RepositoriesFused GPU Kernel Composition

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding. The project supports building hybrid sequence models that interleave standard at

FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat

AITemplate 是一个提前（ahead-of-time）深度学习编译器，将 PyTorch 神经网络转换为独立的 C++ 源代码。它作为一个 PyTorch 转 C++ 编译器和 GPU 内核融合引擎，生成自包含的可执行二进制文件，无需 Python 解释器或深度学习框架运行时即可运行推理。 该项目专门为 NVIDIA TensorCores 和 AMD MatrixCores 生成优化的 CUDA 和 HIP C++ 代码。它专注于通过将多个神经网络算子组合成单个 GPU 内核的系统，最大限度地提高半精度浮点运算的吞吐量，从而最小化内存开销和延迟。 该工具集涵盖了 GPU 推理加速和高性能计算，提供了自定义 GPU 算子开发以及将图节点映射到硬件特定模板的能力。它包括用于基准测试推理性能和可视化模型优化的实用支持。

TNN 是一个深度学习推理框架，旨在跨移动端、桌面端和服务器硬件执行预训练神经网络。它作为硬件加速运行时和模型压缩工具包，为在不同环境中部署模型提供了统一接口。 该框架包含一个 ONNX 模型转换器，可将来自各种训练框架的模型转换为标准化的内部格式。其特色在于结合了模型压缩工具（包括权重量化和静态代码剪枝）以及一种在非依赖节点之间重用缓冲区的内存管理系统，以减少 RAM 使用。 该系统通过算子融合优化性能以最小化内存访问，并采用平台特定的后端来利用专用处理器和 GPU。它还通过低精度计算和硬件特定调优进一步提高了执行速度。

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b