xlite-dev/LeetCUDA
LeetCUDA
LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows.
The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shared memory swizzling to prevent bank conflicts. It also features advanced attention implementations, such as FlashAttention and FlashAttention-2, utilizing fine-grained tiling and low-level assembly.
The library covers a broad surface of GPU primitives, including a variety of activation functions, normalization layers, and matrix operations. It also implements parallel patterns such as warp-level reductions, dot products, and matrix transpositions. For computer vision tasks, it includes a GPU-accelerated implementation of non-maximum suppression.
The repository includes tools for verifying hardware performance through assembly inspection, kernel profiling, and throughput benchmarking against standard libraries.
Features
- GPU Kernel Implementations - Provides a comprehensive collection of custom-written CUDA kernels for accelerated parallel computing and neural network operations.
- Activation Functions - Ships a suite of optimized CUDA kernels for element-wise activation functions like ReLU, GELU, and Sigmoid.
- Attention Kernel Libraries - Provides optimized CUDA kernels for scaled dot-product attention using tensor cores and fine-grained tiling.
- FlashAttention - Implements FlashAttention using low-level assembly and fine-grained tiling for extreme memory optimization.
- FlashAttention-2 - Implements FlashAttention-2 utilizing tensor cores and hardware instructions for high-dimensional tensor operations.
- Half-Precision Matrix Multiplications - Provides half-precision matrix multiplication utilizing shared memory swizzling and asynchronous copy instructions for maximum throughput.
- Normalization Layers - Implements layer normalization to standardize vectors and stabilize training in neural networks.
- Python Bindings - Provides Python bindings that wrap custom CUDA kernels for use within PyTorch deep learning workflows.
- Tensor Core Optimization - Leverages tensor cores for hardware-accelerated half-precision matrix multiplications and tensor operations.
- Matrix Multiplication Utilities - Implements single-precision matrix multiplication optimized with shared memory tiling and double buffering.
- Vectorized Memory Access - Increases throughput by loading multiple data elements simultaneously using vectorized types.
- Asynchronous Data Pipelining - Implements asynchronous data pipelining to overlap global memory loads with computation using double buffering.
- GPU Memory Optimizations - Implements shared memory swizzling, double buffering, and vectorized access to maximize GPU memory throughput.
- C-Bindings - Ships Python wrappers providing direct access to low-level C++ CUDA kernels for performance-critical operations.
- Shared Memory Swizzling - Prevents shared memory bank conflicts by rearranging data access patterns via swizzling.
- Warp-Level Primitives - Implements high-performance warp-level reduction and shuffle instructions for parallel data aggregation on GPUs.
- GPU Matrix Operation Implementations - Provides high-performance GPU implementations of matrix operations utilizing specialized libraries like CUTLASS and CuTe.
- Matrix-Vector Products - Implements single-precision matrix-vector products using tiling strategies and kernel optimizations.
- Vectorized Memory Primitives - Utilizes vectorized memory primitives to load multiple data elements simultaneously and increase global memory throughput.
- Tiled Memory Access Patterns - Implements fine-grained tiling to manage memory usage and maintain constant complexity at the hardware level.
- Attention State Merging - Implements attention state merging for split-KV scenarios to reduce inference latency.
- Activation Functions - Implements the Gaussian Error Linear Unit (GELU) activation using precision types and vectorization.
- Python Bindings - Wraps custom CUDA kernels into Python bindings for seamless integration with PyTorch deep learning workflows.
- Embedding Lookup Kernels - Implements specialized GPU kernels for embedding lookups using precision levels and vectorization.
- Triton Kernels - Implements fused operations and attention states using the Triton domain-specific language.
- Normalization Gradient Computations - Computes vector-jacobian products for weights and biases in layer normalization using parallel reduction.
- Half-Precision Matrix-Vector Operations - Executes half-precision matrix-vector multiplication using tiling strategies and hardware acceleration.
- Online Softmax Kernels - Implements softmax calculations using online techniques for GPU acceleration.
- Rotary Positional Embeddings - Computes rotary positional embeddings to support relative token positioning in language model sequences.
- RMS Normalizations - Provides root mean square layer normalization across various floating-point precisions and vectorization strategies.
- Tensor Reductions - Performs sum reductions across warps and blocks using floating-point and integer precisions.
- Warp-Level Reductions - Implements warp-level reductions using shuffle instructions to aggregate values across threads.
- Element-wise Array Operations - Performs element-wise addition across float32 and float16 arrays using vectorized memory access.
- Matrix Transposition Kernels - Rearranges matrix elements between rows and columns using vectorization and shared memory.
- Online Softmax - Implements an online softmax approach to compute probabilities across tokens while reducing memory overhead.
- Block-Level Reductions - Calculates total sums across a block using warp-level primitives and shared memory.
- Vector Dot Product Kernels - Computes the scalar product of two vectors using floating-point precisions and vectorization.
