flashinfer-ai/flashinfer

Features

• Attention Kernel Libraries - Provides a library of GPU kernels for accelerating prefill, decode, and append phases of transformer attention with paged or ragged KV-caches.
• Batched Prefill Attention Kernels - Computes batched prefill attention using paged key-value caches for efficient prompt processing.
• cuDNN-Accelerated Attention Kernels - Ships cuDNN-accelerated attention kernels for prefill and decode stages in LLM inference.
• Paged Cache Prefill Kernels - Ships GPU kernels for batched prefill attention with paged key-value caches.
• Paged Cache Masking Kernels - Provides GPU kernels for paged cache prefill with causal and custom attention masks.
• Attention State Merging - Merges multiple attention states from separate KV segments using log-sum-exp weighting.
• FP4 Quantization Kernels - Provides GPU kernels for FP4 block-scaled quantization of QKV tensors for attention.
• FP4 Quantized Attention Forward Pass - Executes fused attention on pre-quantized FP4 tensors for fast language model inference.
• FP4 Quantized Forward Passes - Executes fused mixture-of-experts forward passes with 4-bit floating point quantization for efficient LLM inference.
• LLM Inference GPU Kernel Libraries - Provides a library of high-performance GPU kernels for accelerating attention, MoE, and normalization during LLM inference.
• Fused All-Reduce - Provides custom GPU kernels for fused all-reduce with normalization, quantization, and residual addition.
• Out-of-Place All-Reduces - This GPU kernel library performs an out-of-place all-reduce operation on tensors across multiple GPUs using a custom NVLink-optimized kernel.
• Mixed Communication Executions - This GPU kernel library executes collective communication using fused GPU kernels, NCCL fallback, or autotuned mode selection for distributed inference.
• Fused GPU Kernel Composition - Ships fused GPU kernels that combine normalization, all-reduce, and quantization into single operations to reduce memory bandwidth and latency.
• Just-In-Time Kernel Compilers - Provides a just-in-time kernel compilation system that compiles and caches GPU kernels on first invocation, selecting optimal tile sizes and backends.
• Mixture of Experts - Implements efficient mixture-of-experts routing and dispatch with top-k selection and all-to-all communication across GPU ranks for LLM inference.
• Ragged Batch Prefill Kernels - Implements fused batch prefill kernels for variable-length sequences with ragged page tables.
• FP4 Quantized Inference Kernels - Provides fused MoE inference kernels with FP4 quantization for efficient language model decoding.
• Fused MoE GPU Kernels - This GPU kernel library executes a fused mixture-of-experts layer using bfloat16 precision and automatic tile-size tuning for optimal performance.
• Runtime-Profiling Kernel Selections - This GPU kernel library profiles kernel implementations at runtime to select the fastest one for each operation and input shape, caching results for reuse.
• Cache Layout Variants - Provides GPU kernels for multi-head prefill attention with multiple cache layout options.
• Fused Multi-Head Prefill Kernels - Executes fused multi-head prefill attention with optional log-sum-exp output for cascade merging.
• Decode-Phase Latent Attention Kernels - Optimizes decode-phase multi-head latent attention with paged key-value caches for LLM inference.
• Latent Attention GPU Kernels - Provides accelerated GPU kernels for multi-head latent attention with paged caches and FP8 quantization.
• MLA Decode Kernels - Decodes multi-head latent attention from a paged KV cache using optimized GPU kernels.
• Paged Cache Attention Kernels - Runs multi-head attention directly on paged key-value caches in fused GPU kernels.
• Block-Scaled Quantization Kernels - Provides GPU-accelerated block-scaled quantization kernels for FP4 and FP8 formats to reduce memory bandwidth in LLM inference.
• Rank Reductions - This GPU kernel library reduces local shards across tensor-parallel ranks using a high-performance all-reduce primitive with selectable strategies.
• Fused Logits Processing Pipelines - Provides fused GPU kernels for chaining temperature scaling, top-k, top-p, and sampling.
• Paged Key-Value Cache Stores - Implements paged key-value cache storage using non-contiguous memory pages for efficient variable-length sequence handling in transformer inference.
• Fused All-Reduce with Normalization and Quantization - Fuses all-reduce with normalization and quantization into single GPU kernels to reduce latency in multi-GPU inference.
• All-Reduces with Residual, RMSNorm, and Quantization - This GPU kernel library fuses tensor all-reduce across ranks with optional RMSNorm, residual addition, and FP8 quantization to cut bandwidth from separate passes.
• Profiling-Guided Mode Selections - This GPU kernel library performs mixed communication between nodes using either fused kernels with virtual memory and shared memory or collective calls, with automatic mode selection based on profiling.
• Fused Multi-Node All-Reduce - Provides fused all-reduce kernels that combine gradient reduction with normalization and quantization across multi-node GPU clusters for reduced latency.
• CUDA Kernel Compilers - This GPU kernel library generates and compiles high-performance CUDA kernels tailored to specific language model operators and hardware.
• Gated Activation Multiplication Fusion Kernels - Fuses gating activation with multiplication into single GPU kernels for gated feed-forward layers.
• Min-P Probability Filters - Provides fused GPU kernels that filter tokens by a minimum probability threshold relative to the maximum token probability.
• cuDNN-Accelerated Decode Attention - Ships cuDNN-accelerated batched decode attention kernels that use paged key-value caches for high-throughput autoregressive decoding.
• Flexible Layout Attention Kernels - Provides attention kernels that handle multiple KV-cache layouts including ragged tensors and paged tables.
• Log-Sum-Exp Prefill Kernels - Ships GPU kernels for single-request prefill that output log-sum-exp for cascade merging.
• Multi-Level Cascade Attention Kernels - Implements multi-level cascade attention GPU kernels that reuse shared paged KV caches indexed by different indirection arrays across prefix levels.
• Single-Request Prefill Kernels - Provides JIT-compiled GPU kernels for single-request prefill attention with key-value cache.
• Attention Kernel Optimizers - Executes an optimized attention kernel to compute attention for a single query using a key-value cache.
• RMSNorm with FP4 Quantization - Provides fused GPU kernels that combine RMS normalization with FP4 quantization into a single operation.
• NVFP4 Quantized Attention - Provides NVFP4 quantization and attention kernels for SM120 GPU hardware.
• Batched Categorical Sampling Kernels - Ships fused GPU kernels for batched categorical sampling with optional index remapping.
• CUDA Graph Capture Wrappers - This GPU kernel library creates a reusable wrapper that manages persistent stream resources and kernel auto-tuning for replayable CUDA graph captures.
• Deterministic Token Sampling Kernels - Provides deterministic and batched token sampling from logits on GPU.
• Fused MoE GPU Kernels - Provides fused GPU kernels for pre-routed and quantized MoE forward passes with FP8, FP4, and bfloat16 precision.
• Token Dispatch to Expert Ranks - This GPU kernel library dispatches each token's payload to the expert rank assigned to it, enabling parallel execution across experts.
• Fused MoE Forward Pass Kernels - Ships fused GPU kernels for end-to-end mixture-of-experts forward passes with pre-routed tokens.
• FP8 Per-Tensor Scaled Forward Passes - Executes fused MoE forward passes with per-tensor FP8 scaling for efficient inference.
• MXINT4 Block-Scaled Forward Passes - Performs fused MoE forward passes with MXINT4 block-scaled weights for memory-efficient inference.
• Top-K Expert Selection Kernels - Selects top-k experts per token with bias and normalization in a fused GPU kernel.
• Generation Temperature Controls - Provides GPU-accelerated logit temperature scaling kernels that divide logits by a configurable temperature value.
• Concurrent Prefill-Decode Kernels - Launches concurrent prefill and decode in a single GPU kernel to overlap chunked prefills with decode.
• Unified Prefill-Decode Dispatch Kernels - Automatically dispatches prefill and decode per request in a single batched attention kernel.
• GPU-Accelerated Token Sampling - Implements GPU-accelerated categorical sampling from logits with filtering strategies.
• All-Reduces with Residual and RMSNorm - This GPU kernel library combines multi-node all-reduce, residual addition, and RMS normalization into a single fused kernel to reduce latency.
• Customizable Operation Fusions - This GPU kernel library executes a custom all-reduce across a process group, optionally fusing bias, residual, weight, and normalization in a single kernel.
• Residual Addition with RMSNorm - Provides fused GPU kernels that combine residual addition with RMS normalization into a single operation.
• Fused Collective Fusions - This GPU kernel library combines all-reduce with all-gather or reduce-scatter with all-reduce into a single kernel for efficient intra- and inter-node GPU communication.
• Gate-Residual-Normalization Fusion Kernels - Fuses gating, residual addition, layer normalization, and scale-shift into a single GPU kernel.
• Residual LayerNorm with Scale-Shift - Provides fused GPU kernels that combine residual addition, layer normalization, and learned scale or shift into a single operation.
• RMSNorm with SiLU Activation - Provides fused GPU kernels that combine RMS normalization with SiLU activation into a single operation.
• RMSNorm with SiLU Activation Fusion - Provides fused GPU kernels that combine RMS normalization with SiLU activation into a single operation.
• Batched FP8 Multiplications - This GPU kernel library performs batched matrix multiplication on FP8 tensors with per-tensor scaling factors, automatically selecting the optimal GPU backend for the target hardware.
• Block-Scaled FP8 Quantization Kernels - Provides GPU kernels for FP8 block-scaled matrix multiplication, attention, and mixed-precision operations.
• Hopper-Tuned GEMMs - This GPU kernel library performs FP8 block-scaled matrix multiplication on Hopper GPUs, automatically selecting the optimal kernel for small batch sizes.
• MXFP8 GEMMs - This GPU kernel library computes the matrix product of two tensors quantized in MXFP8 block format, supporting multiple backends and scale layouts.
• RMSNorm with FP8 Quantization - Provides fused GPU kernels that combine RMS normalization with FP8 quantization into a single operation.
• Block-Scaled Low-Precision Matrix Operations - Performs matrix multiplication with FP8, FP4, and block-scaled formats to reduce memory and accelerate inference.
• Mixed-Precision GEMMs - This GPU kernel library multiplies two FP8-encoded matrices using a high-performance GPU kernel and returns a bfloat16 tensor.
• MoE Grouped Variants - This GPU kernel library computes expert-partitioned matrix multiplication for mixture-of-experts layers, using FP8 precision with token activations and expert weight matrices.
• Normalization Layers - Provides GPU-accelerated layer normalization kernels that apply scale and bias parameters to stabilize hidden states during inference.
• LoRA-A Projection Kernels - This GPU kernel library projects token activations through expert-specific LoRA A matrices for mixture-of-experts layers.
• Multi-Backend MoE Layers - This GPU kernel library runs a forward pass of a mixture-of-experts layer using optimized CUDA backends such as CUTLASS, TensorRT-LLM, and CuTe-DSL.
• MXINT4 Quantized Inference Kernels - Ships fused MoE inference kernels with MXINT4 block-scaled quantization for efficient language model inference.
• MoE Adapter Applications - This GPU kernel library computes the LoRA delta for mixture-of-experts by fusing top-k routing weights with per-expert LoRA-A and LoRA-B adapters in a single batched call.
• Multi-Adapter Serving Kernels - This GPU kernel library shrinks input through LoRA-A matrices and expands through LoRA-B matrices with routing weights in a single fused BGMV kernel.
• FP8 Block-Scaled - Fuses MoE operations with FP8 block-scaled quantization into single GPU kernels for faster inference.
• Matrix Absorption MLA Kernels - Implements multi-head latent attention with matrix absorption for efficient decode in DeepSeek models.
• Merged Cache MLA Kernels - Computes multi-head latent attention with merged key-value caches and separated RoPE dimensions.
• Nucleus Sampling - Implements nucleus sampling with cumulative probability filtering on GPU.
• Combined Top-K and Top-P Filtering - Provides fused GPU kernels that combine top-k and top-p filtering into a single operation for token sampling.
• Min-P Threshold Sampling - Provides fused GPU kernels that filter and renormalize probabilities by a minimum threshold per token.
• Rotary Embedding Application Kernels - Computes and applies rotary position embeddings in-place on query and key tensors with ragged batching.
• MLA Decode PyTorch Wrappers - Ships a PyTorch wrapper for planning and executing multi-head latent attention decode kernels.
• FP8 Block-Scaled MoE Kernels - Runs MoE layers with FP8 block-scaled quantization to reduce memory and accelerate inference.
• Per-Token Scaled MoE Kernels - This GPU kernel library multiplies groups of rows with corresponding matrices using FP8 quantization and per-token scaling, optimized for mixture-of-experts routing.
• Cascade Attention Kernels - Provides GPU kernels for cascade attention that merge results from shared and per-query segments over a paged KV cache for memory-efficient inference.
• RMS Normalizations - Provides GPU-accelerated root mean square normalization kernels that apply weight scaling and epsilon for numerical stability.
• Conditional RMS Normalization Kernels - Provides fused GPU kernels that apply root mean square normalization with residual addition and learned weight scaling.
• FP4 Quantized Normalization Fusion Kernels - Combines RMS normalization with FP4 quantization into a single fused GPU kernel.
• RMSNorm with Gemma Variants - Provides fused GPU kernels that apply RMS normalization with support for Gemma variants and fp8 quantization.
• Sequence Decoders - Decodes multiple sequences simultaneously using a paged key-value cache to compute attention results for language model inference.
• Sort-Free Sampling Kernels - Implements sort-free top-K, top-P, and min-P sampling on GPU for faster decoding.
• Block-Sparse Attention Kernels - Implements block-sparse masked attention kernels that process only non-zero blocks for efficiency.
• Threshold-Based Softmax Masks - Provides fused GPU kernels for softmax with temperature and top-K/P threshold masking.
• Attention Rescoring Decoders - Decodes a batch of query tokens using cached key-value pairs, computing attention outputs efficiently for language model inference.
• GPU Buffer Allocations - Provides GPU buffer allocation with multicast access for multi-node communication in distributed inference.
• Top-K Constrained Sampling Kernels - Provides fused GPU kernels for top-K constrained sampling via rejection.
• Top-K Index Selection Kernels - This GPU kernel library fuses top-k score selection with page table lookup to map cache positions for sparse attention.
• MoE Top-K Selection Kernels - Selects top-k experts per token with bias and group constraints in fused GPU kernels.
• Radix-Based Top-K Selections - Provides fused GPU kernels that select top-k values using a radix-based algorithm for large tensors.
• Top-K Probability Masks - Ships fused GPU kernels for top-k probability masking to narrow token selection.
• Top-K Probability Renormalization Kernels - Provides fused GPU kernels for top-k probability renormalization in one call.
• Top-P Renormalization Kernels - Provides fused GPU kernels for top-p renormalization after cumulative probability filtering.
• Append Kernels - This GPU kernel library appends a batch of key-value pairs from a ragged tensor into a paged key-value cache, updating page indices and positions.
• Ragged Self-Attention Kernels - Computes self-attention on variable-length sequences using packed tensors and paged KV caches.
• Batched KV Cache Decode Kernels - Provides batched GPU kernels that decode query tokens using cached key-value pairs for efficient LLM inference workflows.
• KV Cache Decode Kernels - Provides GPU kernels that decode query tokens using cached key-value pairs for efficient attention computation during language model inference.
• Paged KV Cache Decode Kernels - Computes attention output for single query tokens using a paged key-value cache during auto-regressive decoding with optimized GPU kernels.
• Attention Kernels - Provides attention kernels with speculative decoding, FP8 quantization, and sliding window support.
• MoE Specializations - Delivers grouped matrix multiplication kernels optimized for MoE expert partitions in LLMs.
• All-Reduces with RMSNorm and Quantization - This GPU kernel library combines all-reduce, residual addition, RMSNorm, and FP8 or FP4 quantization into a single GPU kernel for efficient distributed inference.
• Expert All-Reduce Kernels - This GPU kernel library combines allreduce, residual addition, RMS normalization, and quantization into one kernel to reduce memory traffic and latency during large language model inference.
• LayerNorm with Residual and Gating - Provides fused GPU kernels that apply layer normalization with support for residual, scale or shift, and gating.
• MoE Finalization Fusions - This GPU kernel library fuses all-reduce, residual addition, normalization, and quantization into a single kernel to reduce memory bandwidth and latency.
• Residual Addition with RMSNorm and FP4 Quantization - Provides fused GPU kernels that combine residual addition, RMS normalization, and FP4 quantization into a single operation.
• RMSNorm with FP8 Quantization Fusion - Provides fused GPU kernels that combine residual addition, RMS normalization, and FP8 quantization into a single kernel call.
• Template-Based Code Generators - Generates specialized GPU kernels from parameterized templates at compile time, adapting to operation shapes and data types.
• GPU Parallelism Partitioners - Partitions GPUs into tensor, pipeline, context, and MoE parallelism clusters for distributed inference.
• Tensor Slice Exchanges - This GPU kernel library sends a designated slice of each rank's tensor to every other rank and receives the combined contributions from all peers.
• Multi-Node GPU Memory Managers - Manages GPU memory shared across nodes via direct interconnects for low-latency distributed inference.
• Paged KV Cache Management - Computes attention output for one or more query tokens using a paged key-value cache, reusing auxiliary structures across transformer layers.
• MLA Append Kernels - This GPU kernel library appends new key and value tensors to a paged cache, supporting standard and multi-head latent attention configurations.
• Gate-Residual-LayerNorm Fusion Kernels - Combines gating, residual addition, and layer normalization into a single GPU kernel.
• FP4 Quantized Multiplications - This GPU kernel library performs batched matrix multiplication using packed FP4 data and groupwise scaling for efficient large-model inference.
• MoE Grouped Variants - This GPU kernel library computes grouped matrix multiplication on FP4 quantized data for Mixture-of-Experts layers using a cuDNN backend.
• Softmax Normalization - Applies softmax to logits to produce normalized probability distributions on GPU.
• Temperature-Scaled Softmax Kernels - Computes softmax with temperature scaling in a fused GPU kernel for logits processing.
• Inference Engines - Kernel library optimized for LLM serving performance.
• Inference Frameworks - Kernel library specifically optimized for LLM serving workloads.

Istoric stele