deepseek-ai/DeepGEMM
DeepGEMM
DeepGEMM is a suite of specialized GPU kernels and a just-in-time compiler designed for low-precision matrix operations, Mixture-of-Experts models, and attention processing. It provides a library of high-performance matrix multiplication kernels using FP8 precision to increase compute throughput and reduce memory usage.
The project features a JIT CUDA kernel compiler that generates and loads optimized compute kernels at runtime to eliminate the need for manual compilation during installation. It includes specialized implementations for grouped matrix multiplication that process multiple grouped axes in a single kernel call and fused acceleration kernels for Mixture-of-Experts models that overlap communication with tensor core computation.
The compute engine also provides a logits processor for multi-query attention that functions across both paged and non-paged memory layouts. Its broader capabilities include dense and grouped matrix operations and the fusion of dispatch, linear layers, and activation functions into single kernels.
Features
- FP8 Matrix Multiplication - Provides a library of high-performance matrix multiplication kernels using FP8 precision for deep learning.
- FP8 GEMM Libraries - Provides a suite of high-performance matrix multiplication kernels using FP8 precision.
- Expert Communication Kernels - Provides fused compute kernels for Mixture-of-Experts models that overlap communication with tensor core computation.
- FP8 Scaling - Implements fine-grained FP8 scaling to maintain high precision during low-precision matrix multiplications.
- Just-In-Time Kernel Compilers - Generates and loads optimized compute kernels at runtime to eliminate the need for manual installation compilation.
- Mixture-of-Experts Inference Optimizers - Optimizes Mixture-of-Experts models by fusing dispatch and linear layers to reduce inference latency.
- MoE Kernel Fusions - Combines dispatch, linear layers, and activation functions into a single operation to overlap communication with tensor core compute.
- Tensor Core Optimization - Optimizes matrix operations by mapping them directly to hardware tensor cores using FP8 formats.
- Grouped GEMM - Provides optimized grouped matrix multiplication that processes multiple grouped axes in a single kernel call.
- JIT Kernel Compilers - Features a runtime system that generates and loads optimized GPU compute kernels without manual compilation.
- Kernel Fusion Operations - Fuses dispatch, linear layers, and activation functions into a single kernel to minimize memory overhead.
- Generalized Matrix Multiplications - Performs high-performance dense matrix multiplication using low-precision formats across various memory layouts.
- Multi-Query Attention Scoring - Implements a specialized scoring mechanism for multi-query attention that works across paged and non-paged memory.
- Logit Processors - Includes a specialized compute kernel for calculating multi-query attention logits across different memory layouts.
- Attention Backends - Accelerates multi-query attention by calculating token logits to speed up indexing and inference.
- Attention Optimization - Optimized matrix multiplication kernels for deep learning.
- Inference Kernels - High-throughput FP8 matrix multiplication library for dense and MoE models.
