Awesome GitHub RepositoriesKernel Composition Frameworks

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Cutlass is a collection of C++ templates and Python interfaces for implementing high-performance linear algebra operations on NVIDIA GPUs. It provides a kernel composition framework for designing custom GPU kernels and a mixed-precision tensor library capable of executing operations across diverse data formats, ranging from 64-bit floating point to 4-bit integers. The project features a toolkit for operator fusion that integrates activation functions and bias calculations directly into matrix multiplication kernels to reduce memory passes. It also includes a Python-based domain-specific langu

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding. The project supports building hybrid sequence models that interleave standard at

FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference. It provides optimized implementations for attention operations (including flash attention, page attention, multi-head latent attention, and cascade attention) using paged key-value caches, fused kernel composition, and just-in-time compilation. The library also includes specialized kernels for mixture-of-experts layers, block-scaled low-precision quantization (FP8, FP4), and distributed collective communication. What distinguishes FlashInfer is its fused all-reduce communicat

AITemplate is an ahead-of-time deep learning compiler that translates PyTorch neural networks into standalone C++ source code. It functions as a PyTorch to C++ compiler and a GPU kernel fusion engine, producing self-contained executable binaries that run inference without requiring a Python interpreter or deep learning framework runtime. The project generates optimized CUDA and HIP C++ code specifically for NVIDIA TensorCores and AMD MatrixCores. It focuses on maximizing throughput for half-precision floating-point operations through a system that combines multiple neural network operators in

TNN este un framework de inferență deep learning conceput pentru a executa rețele neuronale pre-antrenate pe hardware mobil, desktop și server. Acesta funcționează ca un runtime accelerat hardware și un toolkit de compresie a modelelor, oferind o interfață unificată pentru implementarea modelelor în medii diverse. Framework-ul include un convertor de modele ONNX pentru a transforma modelele din diverse framework-uri de antrenament într-un format intern standardizat. Se distinge printr-o combinație de instrumente de compresie a modelelor—inclusiv cuantizarea ponderilor și pruning-ul codului static—și un sistem de gestionare a memoriei care reutilizează bufferele între nodurile independente pentru a reduce utilizarea RAM. Sistemul optimizează performanța prin fuziunea operatorilor pentru a minimiza accesul la memorie și utilizează backend-uri specifice platformei pentru a profita de procesoare specializate și GPU-uri. De asemenea, crește viteza de execuție prin calcule de precizie scăzută și ajustări specifice hardware-ului.

This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations. The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

GPyTorch este un framework probabilistic accelerat pe GPU și o bibliotecă PyTorch pentru implementarea modelelor scalabile de procese Gaussian. Oferă un sistem pentru modelarea proceselor Gaussian și estimarea incertitudinii, conceput pentru a efectua operațiuni matriceale eficiente pe hardware grafic. Framework-ul dispune de un sistem modular de kernel pentru construirea funcțiilor de covarianță personalizate și modelarea dependențelor complexe de date. Acesta integrează în mod specific procesele Gaussian cu rețelele neuronale profunde pentru a crea modele hibride pentru regresie și clasificare. Sistemul utilizează tehnici de algebră liniară numerică, inclusiv gradienți conjugați precondiționați și operațiuni bazate pe tensori, pentru a gestiona seturi de date mari. De asemenea, suportă inferența variațională black-box și diferențierea automată pentru optimizarea hiperparametrilor.

Open-Instruct is a distributed training and instruction tuning framework for large language models. It functions as a coordinator for supervised fine-tuning, reinforcement learning from human feedback pipelines, and tool-use training, providing specialized roles for dataset curation and model alignment. The project distinguishes itself through a high-performance training architecture that utilizes actor-based distributed coordination and hybrid sharding to manage large GPU clusters. It implements advanced alignment techniques including direct preference optimization, group relative policy opt

ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized diffusion models within ComfyUI visual workflows. It provides a backend that reduces memory overhead and increases generation speed for transformer models. The project includes specialized tools for identity-preserving generation and an image-to-image guidance toolkit that uses depth maps and reference images. It also features a multimodal visual question answering implementation and a utility for merging multiple quantized model files into single unified files. The engine covers a b