Awesome GitHub RepositoriesFused GPU Kernel Composition

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding. The project supports building hybrid sequence models that interleave standard at

FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat

AITemplate ist ein Ahead-of-Time-Deep-Learning-Compiler, der PyTorch-neuronale Netze in eigenständigen C++-Quellcode übersetzt. Er fungiert als PyTorch-zu-C++-Compiler und GPU-Kernel-Fusion-Engine und erzeugt in sich geschlossene ausführbare Binärdateien, die Inferenz ausführen, ohne einen Python-Interpreter oder eine Deep-Learning-Framework-Runtime zu benötigen. Das Projekt generiert optimierten CUDA- und HIP-C++-Code speziell für NVIDIA TensorCores und AMD MatrixCores. Es konzentriert sich auf die Maximierung des Durchsatzes für Gleitkommaoperationen mit halber Präzision durch ein System, das mehrere neuronale Netzwerkoperatoren zu einzelnen GPU-Kernels kombiniert, um Speicher-Overhead und Latenz zu minimieren. Das Toolset deckt GPU-Inferenzbeschleunigung und High-Performance-Computing ab und bietet Funktionen für die Entwicklung benutzerdefinierter GPU-Operatoren sowie das Mapping von Graph-Knoten auf hardware-spezifische Templates. Es enthält Utility-Unterstützung für das Benchmarking der Inferenz-Performance und die Visualisierung von Modelloptimierungen.

TNN ist ein Deep-Learning-Inferenz-Framework, das für die Ausführung vortrainierter neuronaler Netzwerke auf Mobil-, Desktop- und Server-Hardware entwickelt wurde. Es fungiert als hardwarebeschleunigte Laufzeitumgebung und Toolkit zur Modellkomprimierung und bietet eine einheitliche Schnittstelle für die Bereitstellung von Modellen in verschiedenen Umgebungen. Das Framework enthält einen ONNX-Modellkonverter, um Modelle aus verschiedenen Trainings-Frameworks in ein standardisiertes internes Format zu transformieren. Es zeichnet sich durch eine Kombination von Modellkomprimierungstools aus – einschließlich Gewichtungsquantisierung und Static-Code-Pruning – sowie ein Speichermanagementsystem, das Puffer zwischen nicht-abhängigen Knoten wiederverwendet, um den RAM-Verbrauch zu senken. Das System optimiert die Leistung durch Operator-Fusion, um Speicherzugriffe zu minimieren, und nutzt plattformspezifische Backends, um spezialisierte Prozessoren und GPUs zu nutzen. Es steigert die Ausführungsgeschwindigkeit weiter durch Berechnungen mit niedriger Präzision und hardware-spezifische Optimierungen.

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b