Open-source alternatives to KVQuant

DeepEP is a distributed model accelerator and expert-parallel communication library designed to optimize the training and inference of large-scale neural networks. It provides specialized GPU communication kernels and a remote GPU memory interface to facilitate high-throughput data exchange between hardware nodes. The system utilizes dynamic kernel generation to compile optimized GPU kernels during execution, removing the need for separate installation compilation steps. It implements virtual-lane traffic isolation to prevent interference between different data streams and employs routing met

DeepGEMM is a suite of specialized GPU kernels and a just-in-time compiler designed for low-precision matrix operations, Mixture-of-Experts models, and attention processing. It provides a library of high-performance matrix multiplication kernels using FP8 precision to increase compute throughput and reduce memory usage. The project features a JIT CUDA kernel compiler that generates and loads optimized compute kernels at runtime to eliminate the need for manual compilation during installation. It includes specialized implementations for grouped matrix multiplication that process multiple group

A bidirectional pipeline parallelism algorithm for computation-communication overlap in DeepSeek V3/R1 training.

Expert Parallelism Load Balancer

FlashMLA is an LLM attention kernel library and inference acceleration library providing a collection of high-performance CUDA kernels. It implements multi-head latent attention mechanisms designed to reduce memory overhead and increase throughput during the forward and backward passes of large language model inference. The library utilizes quantized cache attention kernels to improve computation efficiency across both sparse and dense token processing. It specifically optimizes the prefill and decoding phases of model inference through these latent attention implementations. The project cov

Code to reproduce results for the paper "Faster Causal Attention Over Large Sequences Through Sparse Flash Attention"

Landmark Attention: Random-Access Infinite Context Length for Transformers

Flaxformer is a transformer library for primarily NLP and multimodal research at Google. It is used for many NLP research use cases, providing both off-the-shelf BERT and T5 models, and several research projects built on shared components.

Trax is a deep learning framework and hardware-agnostic tensor engine designed for designing and training neural networks. It serves as a research tool providing high-level combinators for composing complex architectures, alongside a dedicated library for building transformer models and a toolkit for reinforcement learning. The framework is distinguished by its support for reversible and sparse transformer architectures, which reduce memory and computational overhead. It enables a single set of model instructions to execute across different hardware backends without changing the underlying co

The repository is the PyTorch implementation of HyperAttention paper:

This repository is the official implementation of INT-FlashAttention.

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs

Large Context Attention

Beyond KV Caching: Shared Attention for Efficient LLMs

NeurIPS'24 Spotlight, ICLR'25, ICML'25 To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

The official repo of MiniMax-Text-01 and MiniMax-VL-01, large-language-model & vision-language-model based on Linear Attention

ICML 2024 Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

Efficient LLM Inference over Long Sequences

Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models

Official implementation of TransNormerLLM: A Faster and Better LLM

The evaluation framework for training-free sparse attention in LLMs

Helpful tools and examples for working with flex-attention

ACL 2025 Squeezed Attention: Accelerating Long Prompt LLM Inference