Awesome GitHub RepositoriesGPU Kernel Implementations

Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

Triton is a parallel computing framework and high-level programming language designed for writing custom compute kernels. It functions as a deep learning compiler, translating complex mathematical operations into high-throughput instructions that maximize hardware utilization and memory efficiency on graphics processing units. The framework distinguishes itself through a hardware-agnostic compute abstraction that allows developers to define kernels without manual low-level tuning. It employs just-in-time compilation to generate optimized binary instructions at runtime, utilizing static data f

This project is a comprehensive computer vision library for the PyTorch ecosystem, providing a standardized collection of neural network architectures, datasets, and high-performance transformation utilities. It serves as a foundational framework for building, training, and deploying deep learning models, offering a centralized model registry that allows developers to instantiate architectures with pre-trained weights for tasks such as image classification, object detection, and semantic segmentation. The library distinguishes itself through its modular approach to data and compute management

Scalene is a high-performance diagnostic utility designed to measure resource consumption during the execution of Python applications. It functions as a line-level monitor, providing granular insights that pinpoint the specific source code responsible for performance overhead. The tool distinguishes itself through statistical profiling that captures stack traces and resource usage without requiring manual instrumentation of the source code. It tracks CPU, GPU, and memory consumption by intercepting library-level calls and hardware driver commands, allowing for the analysis of both managed and

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Taskflow is a C++ task-parallel framework designed to build high-performance parallel workflows and complex dependency graphs. It provides a programming model that organizes computational work into directed acyclic graphs, enabling developers to manage concurrency, resource scheduling, and task dependencies across multi-core CPUs and GPU accelerators. The framework distinguishes itself through its ability to orchestrate heterogeneous systems, allowing for the integration of hardware-accelerated kernels and memory operations into unified execution pipelines. It supports dynamic runtime subflow

CuPy هي مكتبة حوسبة مصفوفات CUDA تنفذ واجهة متوافقة مع NumPy لتنفيذ عمليات المصفوفات والحوسبة الرقمية على وحدات معالجة الرسومات من NVIDIA. تعمل كمكتبة رقمية مسرعة بواسطة GPU وتنفيذ لـ SciPy يعتمد على CUDA، مما يفرغ الحسابات الثقيلة إلى أجهزة الرسومات لزيادة سرعة المعالجة لأعباء العمل العلمية والهندسية. تمكّن المكتبة تبادل المصفوفات بين أطر عمل متعددة، مما يسمح بمشاركة مخازن البيانات بين أطر عمل التعلم العميق المختلفة باستخدام تخطيطات ذاكرة موحدة لتجنب نسخ الذاكرة. كما تدعم دمج نواة GPU مخصصة، مما يسمح بربط بيانات المصفوفات بواجهات برمجة تطبيقات منخفضة المستوى للتحكم الدقيق في تنفيذ الأجهزة. بشكل عام، يغطي المشروع سير عمل معالجة المصفوفات والحوسبة العلمية عالية الأداء. تشمل قدراته تسريع حسابات المصفوفات وتوفير أدوات للحسابات الرقمية واسعة النطاق.

Numba هو مترجم فوري (JIT) يترجم دوال Python عالية المستوى إلى كود آلة محسن في وقت التشغيل. من خلال الاستفادة من بنية المترجم LLVM، فإنه يوفر إطار عمل لتسريع معالجة البيانات الرقمية والحسابات الرياضية، مما يتيح مستويات أداء تضاهي اللغات المجمعة استاتيكياً. يتميز المشروع بقدرته على إجراء التخصيص القائم على استنتاج النوع، والذي يولد تعليمات آلة مصممة خصيصاً لأنواع البيانات المستخدمة أثناء التنفيذ. يستخدم خط تجميع كسول يؤجل الترجمة حتى لحظة الاستدعاء، مما يقلل من عبء بدء التشغيل مع الحفاظ على أداء ثابت عبر معماريات المعالجات وأنظمة التشغيل المختلفة. إلى جانب التجميع الأساسي، توفر مجموعة الأدوات دعماً واسعاً لتسريع الأجهزة من خلال توزيع العمليات التكرارية وتعبيرات المصفوفات عبر أنوية CPU متعددة ووحدات معالجة الرسومات. تستخدم استراتيجيات التوجيه (vectorization) والتوازي لزيادة الإنتاجية لمجموعات البيانات الرقمية واسعة النطاق، مما يسمح للمطورين باستهداف الأجهزة المتخصصة مباشرة من الكود القياسي.

xformers is a collection of specialized toolsets for fused GPU operators, sparse attention mechanisms, modular transformer components, and performance benchmarking. It provides a library of optimized and interoperable building blocks used to construct and experiment with transformer architectures. The project features a fused CUDA operator library that combines common layers into single GPU operations to increase throughput. It includes a sparse attention framework and memory-efficient attention kernels that utilize tiling strategies and structured sparsity patterns to reduce computational ov

Cutlass is a collection of C++ templates and Python interfaces for implementing high-performance linear algebra operations on NVIDIA GPUs. It provides a kernel composition framework for designing custom GPU kernels and a mixed-precision tensor library capable of executing operations across diverse data formats, ranging from 64-bit floating point to 4-bit integers. The project features a toolkit for operator fusion that integrates activation functions and bias calculations directly into matrix multiplication kernels to reduce memory passes. It also includes a Python-based domain-specific langu

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput. The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed. The too

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

bitsandbytes is a quantization library for large language models that reduces memory footprints using k-bit quantization. It provides a framework for 4-bit low-rank adaptation, tools for 8-bit model compression, and memory-efficient optimizer extensions for PyTorch. The project enables the training of large models on limited hardware through 4-bit quantization and low-rank adaptation weights. It also facilitates faster inference by compressing models to 8-bit precision using vector-wise quantization. The library covers a range of memory optimization capabilities, including optimizer memory r

SD.Next is an all-in-one web interface and multi-backend inference engine for generating, editing, and processing images and videos using diffusion models. It functions as a comprehensive tool for diffusion model management and an automated image processing pipeline for bulk operations. The project is distinguished by its hardware-backend abstraction layer, which provides automatic detection and acceleration for NVIDIA CUDA, AMD ROCm, Intel OpenVINO, and DirectML. It features a headless generative API and a programmatic command interface, allowing users to trigger tasks via REST API or CLI wi

Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core opera

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes. The project distinguishes itself through its ability to perform runtime m