linkedin/Liger-Kernel
Liger Kernel
Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes.
The project distinguishes itself through its ability to perform runtime model surgery via monkey-patching, allowing specific layers in Hugging Face or Megatron-LM models to be swapped for optimized kernels without altering source code. It also offers fused kernel techniques that combine multiple operations into single GPU kernels, including preference optimization loss fusion that reduces memory usage during fine-tuning by up to 80%. Multi-stream residual wrapping stabilizes deep network training by wrapping arbitrary layers with doubly-stochastic residual streams.
The kernel library covers a broad range of operations including RMS layer normalization, rotary position embeddings, softmax and sparsemax computation, multi-token attention, and fused linear cross-entropy. It supports alignment loss computation for methods such as DPO, ORPO, SimPO, and CPO, as well as distillation loss computation for knowledge distillation tasks. The project integrates with distributed training frameworks including FSDP, DeepSpeed, and DDP without additional configuration.
Features
- Triton Kernels - Provides a collection of pre-built fused Triton kernels that combine multiple LLM operations into single GPU kernels.
- Transformer Training Accelerators - Provides GPU-optimized Triton kernels that accelerate transformer training throughput by up to 20%.
- Bitwise-Exact Gradient Kernels - Guarantees bitwise-identical gradients when replacing standard operations with fused kernels.
- Preference Optimization Losses - Calculates fused linear losses for preference optimization methods including DPO, ORPO, SimPO, and CPO.
- Fused Linear Alignment Losses - Fused linear transformations with alignment loss calculations to reduce memory usage during fine-tuning by up to 80%.
- Preference Optimization - Computes fused linear losses for alignment methods such as DPO, ORPO, SimPO, and CPO to reduce memory usage during fine-tuning.
- Training Memory Optimizers - Reduces LLM training memory footprint through efficient fused kernel implementations that minimize intermediate storage.
- Alignment Loss Optimizers - Provides fused kernels that cut memory usage by up to 80% during preference alignment fine-tuning.
- Drop-In Kernel Optimizers - Provides drop-in kernel replacements for Hugging Face transformer models that increase throughput and reduce memory without changing model code.
- Layer Optimization Patches - Optimizes Hugging Face transformer models by swapping standard layers for memory-efficient Triton kernels with a single function call.
- Runtime Kernel Swaps - Swaps standard Hugging Face model layers for optimized Triton kernels with a single function call, preserving exact computation.
- Gradient-Preserving Fusions - Provides fused kernels that guarantee bitwise-exact gradients, enabling safe drop-in replacement of standard operations.
- Model Layer Swapping Patches - Swaps Hugging Face model layers for optimized kernels via monkey-patching with a single function call.
- Model Layer Patching - Enables runtime monkey-patching of Hugging Face and Megatron-LM model layers with optimized Triton kernels.
- Multi-Query Attention Scoring - Ships an optimized fused multi-token attention kernel for multiple query and key inputs.
- Megatron-LM Kernel Patches - Replaces RMSNorm and cross-entropy loss in Megatron-LM with faster Triton kernels.
- Megatron-LM Patches - Provides optimized Triton kernel replacements for Megatron-LM training setups targeting RMSNorm and cross-entropy loss operations.
- Divergence Loss Kernels - Provides optimized kernel implementations for calculating KL divergence and Jensen-Shannon divergence losses for knowledge distillation.
- Distillation Loss Calculators - Implements optimized KL divergence and Jensen-Shannon divergence kernels for knowledge distillation.
- Layer-Wise Model Assembly - Ships individual optimized fused kernels that can be imported and combined as modular building blocks for custom architectures.
- Modular Layer Compositions - Offers individual optimized fused kernels that can be imported and combined into user-defined neural network architectures.
- Multi-GPU Training Utilities - Integrates with FSDP, DeepSpeed, and DDP for multi-GPU training without additional configuration.
- Rotary Positional Embeddings - Implements an optimized rotary position embedding kernel for transformer models.
- Cross-Entropy Loss Functions - Ships an optimized fused cross-entropy loss kernel for large-vocabulary classification tasks.
- Fused Linear Cross-Entropy Kernels - Combines linear transformation and cross-entropy loss into a single fused GPU kernel.
- Doubly-Stochastic Residual Streams - Implements multi-stream residual wrapping with doubly-stochastic matrices to stabilize deep network training.
- RMS Normalizations - Provides an optimized fused RMS normalization kernel for transformer layers.
- Megatron-LM Kernel Patches - Accelerates Megatron-LM training by patching RMSNorm and cross-entropy loss with faster, more memory-efficient Triton kernels.
- Softmax Normalization - Ships an optimized softmax kernel that converts raw scores to probability distributions.
- Fine-Tuning Frameworks - Efficient Triton kernels for optimized model training.
- Inference and Serving - Efficient Triton kernels for optimized LLM training and inference.
- Computation and Optimization - Triton kernels optimized for large language model training.
- Fine-Tuning Frameworks - Efficient Triton kernels for training optimization.
