Awesome GitHub RepositoriesGPU Kernel Implementations

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Triton is a parallel computing framework and high-level programming language designed for writing custom compute kernels. It functions as a deep learning compiler, translating complex mathematical operations into high-throughput instructions that maximize hardware utilization and memory efficiency on graphics processing units. The framework distinguishes itself through a hardware-agnostic compute abstraction that allows developers to define kernels without manual low-level tuning. It employs just-in-time compilation to generate optimized binary instructions at runtime, utilizing static data f

This project is a comprehensive computer vision library for the PyTorch ecosystem, providing a standardized collection of neural network architectures, datasets, and high-performance transformation utilities. It serves as a foundational framework for building, training, and deploying deep learning models, offering a centralized model registry that allows developers to instantiate architectures with pre-trained weights for tasks such as image classification, object detection, and semantic segmentation. The library distinguishes itself through its modular approach to data and compute management

Scalene is a high-performance diagnostic utility designed to measure resource consumption during the execution of Python applications. It functions as a line-level monitor, providing granular insights that pinpoint the specific source code responsible for performance overhead. The tool distinguishes itself through statistical profiling that captures stack traces and resource usage without requiring manual instrumentation of the source code. It tracks CPU, GPU, and memory consumption by intercepting library-level calls and hardware driver commands, allowing for the analysis of both managed and

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

CuPy es una biblioteca de computación de matrices CUDA que implementa una interfaz compatible con NumPy para ejecutar operaciones de matrices y computación numérica en GPUs NVIDIA. Sirve como una biblioteca numérica acelerada por GPU y una implementación de SciPy basada en CUDA, descargando cálculos pesados al hardware gráfico para aumentar la velocidad de procesamiento para cargas de trabajo científicas y de ingeniería. La biblioteca permite el intercambio de tensores entre múltiples frameworks, permitiendo que los búferes de datos se compartan entre diferentes frameworks de aprendizaje profundo utilizando diseños de memoria estandarizados para evitar copias de memoria. También admite la integración de kernels de GPU personalizados, permitiendo que los datos de las matrices se conecten a APIs de bajo nivel para un control preciso sobre la ejecución del hardware. En términos generales, el proyecto cubre flujos de trabajo de procesamiento de matrices y computación científica de alto rendimiento. Sus capacidades incluyen la aceleración de cálculos de matrices y la provisión de herramientas para cálculos numéricos a gran escala.

Numba es un compilador just-in-time que traduce funciones de Python de alto nivel a código máquina optimizado en tiempo de ejecución. Al aprovechar la infraestructura del compilador LLVM, proporciona un marco para acelerar el procesamiento de datos numéricos y los cálculos matemáticos, permitiendo niveles de rendimiento comparables a los lenguajes compilados estáticamente. El proyecto se distingue por su capacidad para realizar especialización basada en inferencia de tipos, lo que genera instrucciones de máquina adaptadas a los tipos de datos específicos utilizados durante la ejecución. Emplea una tubería de compilación perezosa que difiere la traducción hasta el momento de la invocación, minimizando la sobrecarga de inicio mientras mantiene un rendimiento consistente en diversas arquitecturas de procesador y sistemas operativos. Más allá de la compilación central, el kit de herramientas proporciona un amplio soporte para la aceleración de hardware mediante la distribución de operaciones iterativas y expresiones de matriz a través de múltiples núcleos de CPU y unidades de procesamiento gráfico. Utiliza estrategias de vectorización y paralelización para maximizar el rendimiento de grandes conjuntos de datos numéricos, permitiendo a los desarrolladores apuntar a hardware especializado directamente desde código estándar.

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Cutlass is a collection of C++ templates and Python interfaces for implementing high-performance linear algebra operations on NVIDIA GPUs. It provides a kernel composition framework for designing custom GPU kernels and a mixed-precision tensor library capable of executing operations across diverse data formats, ranging from 64-bit floating point to 4-bit integers. The project features a toolkit for operator fusion that integrates activation functions and bias calculations directly into matrix multiplication kernels to reduce memory passes. It also includes a Python-based domain-specific langu

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

bitsandbytes is a quantization library for large language models that reduces memory footprints using k-bit quantization. It provides a framework for 4-bit low-rank adaptation, tools for 8-bit model compression, and memory-efficient optimizer extensions for PyTorch. The project enables the training of large models on limited hardware through 4-bit quantization and low-rank adaptation weights. It also facilitates faster inference by compressing models to 8-bit precision using vector-wise quantization. The library covers a range of memory optimization capabilities, including optimizer memory r

SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m