awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
linkedin avatar

linkedin/Liger-Kernel

0
View on GitHub↗
linkedin.github.io/Liger-Kernel↗

Liger Kernel

Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching pre-existing models with minimal code changes.

The project distinguishes itself through its ability to perform runtime model surgery via monkey-patching, allowing specific layers in Hugging Face or Megatron-LM models to be swapped for optimized kernels without altering source code. It also offers fused kernel techniques that combine multiple operations into single GPU kernels, including preference optimization loss fusion that reduces memory usage during fine-tuning by up to 80%. Multi-stream residual wrapping stabilizes deep network training by wrapping arbitrary layers with doubly-stochastic residual streams.

The kernel library covers a broad range of operations including RMS layer normalization, rotary position embeddings, softmax and sparsemax computation, multi-token attention, and fused linear cross-entropy. It supports alignment loss computation for methods such as DPO, ORPO, SimPO, and CPO, as well as distillation loss computation for knowledge distillation tasks. The project integrates with distributed training frameworks including FSDP, DeepSpeed, and DDP without additional configuration.

Features

  • Triton Kernels - Provides a collection of pre-built fused Triton kernels that combine multiple LLM operations into single GPU kernels.
  • Transformer Training Accelerators - Provides GPU-optimized Triton kernels that accelerate transformer training throughput by up to 20%.
  • Bitwise-Exact Gradient Kernels - Guarantees bitwise-identical gradients when replacing standard operations with fused kernels.
  • Preference Optimization Losses - Calculates fused linear losses for preference optimization methods including DPO, ORPO, SimPO, and CPO.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
6,148 stars·489 forks·Python·bsd-2-clause·37 views
  • Fused Linear Alignment Losses - Fused linear transformations with alignment loss calculations to reduce memory usage during fine-tuning by up to 80%.
  • Preference Optimization - Computes fused linear losses for alignment methods such as DPO, ORPO, SimPO, and CPO to reduce memory usage during fine-tuning.
  • Training Memory Optimizers - Reduces LLM training memory footprint through efficient fused kernel implementations that minimize intermediate storage.
  • Alignment Loss Optimizers - Provides fused kernels that cut memory usage by up to 80% during preference alignment fine-tuning.
  • Drop-In Kernel Optimizers - Provides drop-in kernel replacements for Hugging Face transformer models that increase throughput and reduce memory without changing model code.
  • Layer Optimization Patches - Optimizes Hugging Face transformer models by swapping standard layers for memory-efficient Triton kernels with a single function call.
  • Runtime Kernel Swaps - Swaps standard Hugging Face model layers for optimized Triton kernels with a single function call, preserving exact computation.
  • Gradient-Preserving Fusions - Provides fused kernels that guarantee bitwise-exact gradients, enabling safe drop-in replacement of standard operations.
  • Model Layer Swapping Patches - Swaps Hugging Face model layers for optimized kernels via monkey-patching with a single function call.
  • Model Layer Patching - Enables runtime monkey-patching of Hugging Face and Megatron-LM model layers with optimized Triton kernels.
  • Multi-Query Attention Scoring - Ships an optimized fused multi-token attention kernel for multiple query and key inputs.
  • Megatron-LM Kernel Patches - Replaces RMSNorm and cross-entropy loss in Megatron-LM with faster Triton kernels.
  • Megatron-LM Patches - Provides optimized Triton kernel replacements for Megatron-LM training setups targeting RMSNorm and cross-entropy loss operations.
  • Divergence Loss Kernels - Provides optimized kernel implementations for calculating KL divergence and Jensen-Shannon divergence losses for knowledge distillation.
  • Distillation Loss Calculators - Implements optimized KL divergence and Jensen-Shannon divergence kernels for knowledge distillation.
  • Layer-Wise Model Assembly - Ships individual optimized fused kernels that can be imported and combined as modular building blocks for custom architectures.
  • Modular Layer Compositions - Offers individual optimized fused kernels that can be imported and combined into user-defined neural network architectures.
  • Multi-GPU Training Utilities - Integrates with FSDP, DeepSpeed, and DDP for multi-GPU training without additional configuration.
  • Rotary Positional Embeddings - Implements an optimized rotary position embedding kernel for transformer models.
  • Cross-Entropy Loss Functions - Ships an optimized fused cross-entropy loss kernel for large-vocabulary classification tasks.
  • Fused Linear Cross-Entropy Kernels - Combines linear transformation and cross-entropy loss into a single fused GPU kernel.
  • Doubly-Stochastic Residual Streams - Implements multi-stream residual wrapping with doubly-stochastic matrices to stabilize deep network training.
  • RMS Normalizations - Provides an optimized fused RMS normalization kernel for transformer layers.
  • Megatron-LM Kernel Patches - Accelerates Megatron-LM training by patching RMSNorm and cross-entropy loss with faster, more memory-efficient Triton kernels.
  • Softmax Normalization - Ships an optimized softmax kernel that converts raw scores to probability distributions.
  • Fine-Tuning Frameworks - Efficient Triton kernels for optimized model training.
  • Inference and Serving - Efficient Triton kernels for optimized LLM training and inference.
  • Computation and Optimization - Triton kernels optimized for large language model training.
  • Fine-Tuning Frameworks - Efficient Triton kernels for training optimization.

    • Star history

    Star history chart for linkedin/liger-kernelStar history chart for linkedin/liger-kernel

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Frequently asked questions

    What does linkedin/liger-kernel do?

    Liger-Kernel is a collection of pre-built fused Triton kernels and patching utilities designed to accelerate large language model training. It provides drop-in kernel replacements for common LLM operations such as RMSNorm, cross-entropy loss, and attention, enabling increased throughput and reduced memory usage while preserving bitwise-exact gradients. The project serves as a toolkit for composing custom model architectures from individual optimized kernels and for patching…

    What are the main features of linkedin/liger-kernel?

    The main features of linkedin/liger-kernel are: Triton Kernels, Transformer Training Accelerators, Bitwise-Exact Gradient Kernels, Preference Optimization Losses, Fused Linear Alignment Losses, Preference Optimization, Training Memory Optimizers, Alignment Loss Optimizers.

    Which projects share features with linkedin/liger-kernel?

    Projects with overlapping indexed features include: internlm/xtuner — xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised… xlite-dev/leetcuda — LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation… pytorch/torchtune — Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a… modelscope/swift — Swift is a toolkit for the full-parameter and parameter-efficient fine-tuning of large language and multimodal models.… flashinfer-ai/flashinfer — FlashInfer is a library of high-performance GPU kernels purpose-built for accelerating large language model inference.… mosaicml/llm-foundry — llm-foundry is a training framework for large language models, providing a system for foundation model pre-training…

    Projects sharing features with Liger Kernel

    These projects share indexed features with Liger Kernel. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • internlm/xtunerInternLM avatar

      InternLM/xtuner

      5,150View on GitHub↗

      xtuner is a comprehensive training engine for large language models, offering a toolkit for pre-training, supervised fine-tuning, and the optimization of vision-language multimodal models. It serves as a distributed training accelerator and a specialized framework for scaling Mixture-of-Experts models and aligning model behavior through reinforcement learning from human feedback. The project distinguishes itself through advanced memory and compute optimizations, such as sequence parallelism for ultra-long context windows and interleaved pipeline parallelism to reduce GPU idle time. It provide

      Pythonagentdeepseek-v3gpt-oss
      View on GitHub↗5,150
    • xlite-dev/leetcudaxlite-dev avatar

      xlite-dev/LeetCUDA

      9,694View on GitHub↗

      LeetCUDA is a collection of high-performance GPU kernel libraries focusing on memory optimization, activation functions, and attention mechanisms. It serves as a reference library for CUDA kernel implementations, ranging from basic element-wise operations to complex neural network components, and provides Python bindings to integrate these kernels into deep learning workflows. The project is distinguished by its focus on low-level hardware optimizations. This includes the use of tensor cores for half-precision matrix multiplication, asynchronous data pipelining with double buffering, and shar

      Cudacudacuda-12cuda-cpp
      View on GitHub↗9,694
    • modelscope/swiftmodelscope avatar

      modelscope/swift

      14,633View on GitHub↗

      Swift is a toolkit for the full-parameter and parameter-efficient fine-tuning of large language and multimodal models. It functions as a multimodal model trainer for text, image, video, and audio data, and includes specialized tools for model compression and reinforcement learning from human feedback. The framework provides an alignment toolkit for optimizing model behavior using preference learning algorithms and reinforcement learning. It integrates parameter-efficient fine-tuning methods to adapt models with minimal memory and compute requirements, alongside utilities for reducing hardware

      Python
      View on GitHub↗14,633
    • pytorch/torchtunepytorch avatar

      pytorch/torchtune

      5,774View on GitHub↗

      Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

      Python
      View on GitHub↗5,774
    Compare all 30 related projects→