Awesome GitHub RepositoriesGPU Kernel Implementations

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Triton is a parallel computing framework and high-level programming language designed for writing custom compute kernels. It functions as a deep learning compiler, translating complex mathematical operations into high-throughput instructions that maximize hardware utilization and memory efficiency on graphics processing units. The framework distinguishes itself through a hardware-agnostic compute abstraction that allows developers to define kernels without manual low-level tuning. It employs just-in-time compilation to generate optimized binary instructions at runtime, utilizing static data f

This project is a comprehensive computer vision library for the PyTorch ecosystem, providing a standardized collection of neural network architectures, datasets, and high-performance transformation utilities. It serves as a foundational framework for building, training, and deploying deep learning models, offering a centralized model registry that allows developers to instantiate architectures with pre-trained weights for tasks such as image classification, object detection, and semantic segmentation. The library distinguishes itself through its modular approach to data and compute management

Scalene is a high-performance diagnostic utility designed to measure resource consumption during the execution of Python applications. It functions as a line-level monitor, providing granular insights that pinpoint the specific source code responsible for performance overhead. The tool distinguishes itself through statistical profiling that captures stack traces and resource usage without requiring manual instrumentation of the source code. It tracks CPU, GPU, and memory consumption by intercepting library-level calls and hardware driver commands, allowing for the analysis of both managed and

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

CuPy ist eine CUDA-Array-Computing-Bibliothek, die eine NumPy-kompatible Schnittstelle für die Ausführung von Array-Operationen und numerischen Berechnungen auf NVIDIA GPUs implementiert. Sie dient als GPU-beschleunigte numerische Bibliothek und CUDA-basierte SciPy-Implementierung, die rechenintensive Aufgaben auf Grafikhardware auslagert, um die Verarbeitungsgeschwindigkeit für wissenschaftliche und technische Workloads zu erhöhen. Die Bibliothek ermöglicht den Austausch von Tensoren zwischen verschiedenen Frameworks, wodurch Datenpuffer zwischen verschiedenen Deep-Learning-Frameworks unter Verwendung standardisierter Speicherlayouts geteilt werden können, um Speicherkopien zu vermeiden. Sie unterstützt zudem die Integration benutzerdefinierter GPU-Kernel, wodurch Array-Daten mit Low-Level-APIs verbunden werden können, um eine präzise Kontrolle über die Hardwareausführung zu ermöglichen. Das Projekt deckt im Wesentlichen Workflows für Array-Verarbeitung und wissenschaftliches Rechnen mit hoher Leistung ab. Zu den Fähigkeiten gehören die Beschleunigung von Array-Berechnungen und die Bereitstellung von Werkzeugen für numerische Berechnungen im großen Maßstab.

Numba ist ein Just-in-Time-Compiler, der hochsprachliche Python-Funktionen zur Laufzeit in optimierten Maschinencode übersetzt. Durch die Nutzung der LLVM-Compiler-Infrastruktur bietet es ein Framework zur Beschleunigung numerischer Datenverarbeitung und mathematischer Berechnungen, was Leistungsniveaus ermöglicht, die mit statisch kompilierten Sprachen vergleichbar sind. Das Projekt zeichnet sich durch seine Fähigkeit zur typinferenzbasierten Spezialisierung aus, die Maschinenbefehle generiert, welche auf die während der Ausführung verwendeten Datentypen zugeschnitten sind. Es verwendet eine Lazy-Compilation-Pipeline, die die Übersetzung bis zum Moment des Aufrufs verzögert, wodurch der Start-Overhead minimiert und eine konsistente Leistung über verschiedene Prozessorarchitekturen und Betriebssysteme hinweg beibehalten wird. Über die reine Kompilierung hinaus bietet das Toolkit umfassende Unterstützung für Hardwarebeschleunigung durch die Verteilung iterativer Operationen und Array-Ausdrücke auf mehrere CPU-Kerne und Grafikprozessoren. Es nutzt Vektorisierungs- und Parallelisierungsstrategien, um den Durchsatz für große numerische Datensätze zu maximieren, wodurch Entwickler spezialisierte Hardware direkt aus Standardcode ansprechen können.

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Cutlass is a collection of C++ templates and Python interfaces for implementing high-performance linear algebra operations on NVIDIA GPUs. It provides a kernel composition framework for designing custom GPU kernels and a mixed-precision tensor library capable of executing operations across diverse data formats, ranging from 64-bit floating point to 4-bit integers. The project features a toolkit for operator fusion that integrates activation functions and bias calculations directly into matrix multiplication kernels to reduce memory passes. It also includes a Python-based domain-specific langu

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

bitsandbytes is a quantization library for large language models that reduces memory footprints using k-bit quantization. It provides a framework for 4-bit low-rank adaptation, tools for 8-bit model compression, and memory-efficient optimizer extensions for PyTorch. The project enables the training of large models on limited hardware through 4-bit quantization and low-rank adaptation weights. It also facilitates faster inference by compressing models to 8-bit precision using vector-wise quantization. The library covers a range of memory optimization capabilities, including optimizer memory r

SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m