Awesome GitHub RepositoriesGPU Kernel Implementations

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Triton is a parallel computing framework and high-level programming language designed for writing custom compute kernels. It functions as a deep learning compiler, translating complex mathematical operations into high-throughput instructions that maximize hardware utilization and memory efficiency on graphics processing units. The framework distinguishes itself through a hardware-agnostic compute abstraction that allows developers to define kernels without manual low-level tuning. It employs just-in-time compilation to generate optimized binary instructions at runtime, utilizing static data f

This project is a comprehensive computer vision library for the PyTorch ecosystem, providing a standardized collection of neural network architectures, datasets, and high-performance transformation utilities. It serves as a foundational framework for building, training, and deploying deep learning models, offering a centralized model registry that allows developers to instantiate architectures with pre-trained weights for tasks such as image classification, object detection, and semantic segmentation. The library distinguishes itself through its modular approach to data and compute management

Scalene is a high-performance diagnostic utility designed to measure resource consumption during the execution of Python applications. It functions as a line-level monitor, providing granular insights that pinpoint the specific source code responsible for performance overhead. The tool distinguishes itself through statistical profiling that captures stack traces and resource usage without requiring manual instrumentation of the source code. It tracks CPU, GPU, and memory consumption by intercepting library-level calls and hardware driver commands, allowing for the analysis of both managed and

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

CuPy este o bibliotecă de calcul array CUDA care implementează o interfață compatibilă cu NumPy pentru executarea operațiunilor pe array-uri și calcul numeric pe GPU-uri NVIDIA. Acesta servește ca bibliotecă numerică accelerată GPU și o implementare SciPy bazată pe CUDA, descărcând calculele grele pe hardware-ul grafic pentru a crește viteza de procesare pentru sarcinile de lucru științifice și inginerești. Biblioteca permite schimbul de tensori între framework-uri, permițând partajarea bufferelor de date între diferite framework-uri de deep learning folosind layout-uri de memorie standardizate pentru a evita copiile de memorie. De asemenea, suportă integrarea kernel-urilor GPU personalizate, permițând conectarea datelor de tip array la API-uri de nivel scăzut pentru un control precis asupra execuției hardware. În linii mari, proiectul acoperă fluxuri de lucru de procesare de array-uri de înaltă performanță și calcul științific. Capabilitățile sale includ accelerarea calculelor pe array-uri și furnizarea de instrumente pentru calcule numerice la scară largă.

Numba este un compilator just-in-time care traduce funcțiile Python de nivel înalt în cod mașină optimizat la runtime. Prin utilizarea infrastructurii de compilare LLVM, acesta oferă un framework pentru accelerarea procesării datelor numerice și a calculelor matematice, permițând niveluri de performanță comparabile cu limbajele compilate static. Proiectul se distinge prin capacitatea sa de a efectua specializarea bazată pe inferența de tipuri, care generează instrucțiuni mașină adaptate tipurilor de date specifice utilizate în timpul execuției. Acesta folosește un pipeline de compilare leneșă (lazy) care amână traducerea până în momentul invocării, minimizând timpul de pornire și menținând o performanță consistentă pe diverse arhitecturi de procesoare și sisteme de operare. Dincolo de compilarea de bază, toolkit-ul oferă suport extins pentru accelerarea hardware prin distribuirea operațiunilor iterative și a expresiilor de tip array pe mai multe nuclee CPU și unități de procesare grafică. Utilizează strategii de vectorizare și paralelizare pentru a maximiza debitul pentru seturi de date numerice la scară largă, permițând dezvoltatorilor să vizeze hardware specializat direct din codul standard.

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Cutlass is a collection of C++ templates and Python interfaces for implementing high-performance linear algebra operations on NVIDIA GPUs. It provides a kernel composition framework for designing custom GPU kernels and a mixed-precision tensor library capable of executing operations across diverse data formats, ranging from 64-bit floating point to 4-bit integers. The project features a toolkit for operator fusion that integrates activation functions and bias calculations directly into matrix multiplication kernels to reduce memory passes. It also includes a Python-based domain-specific langu

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

bitsandbytes is a quantization library for large language models that reduces memory footprints using k-bit quantization. It provides a framework for 4-bit low-rank adaptation, tools for 8-bit model compression, and memory-efficient optimizer extensions for PyTorch. The project enables the training of large models on limited hardware through 4-bit quantization and low-rank adaptation weights. It also facilitates faster inference by compressing models to 8-bit precision using vector-wise quantization. The library covers a range of memory optimization capabilities, including optimizer memory r

SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m