# fla-org/flash-linear-attention

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/fla-org-flash-linear-attention).**

4,406 stars · 390 forks · Python · mit

## Links

- GitHub: https://github.com/fla-org/flash-linear-attention
- Homepage: https://github.com/fla-org/flash-linear-attention
- awesome-repositories: https://awesome-repositories.com/repository/fla-org-flash-linear-attention.md

## Topics

`large-language-models` `machine-learning-systems` `natural-language-processing`

## Description

Flash Linear Attention is a training framework and inference engine for sequence models that use linear attention and state space mechanisms, designed to process long contexts with reduced memory and compute overhead. It provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures, and includes a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding.

The project supports building hybrid sequence models that interleave standard attention with linear attention and state space layers, balancing efficiency with global context. It includes a distributed checkpoint manager that splits model weights across multiple files for parallel loading and saving in multi-node training, and a weight format transpilation utility for converting between Hugging Face and distributed checkpoint formats. The framework also provides hardware-aware kernel dispatch that selects optimized CUDA kernels at runtime based on GPU architecture and tensor shapes.

The training surface covers training models from scratch, continuing pretraining from checkpoints, launching multi-node training, and automatically resuming interrupted training from the last saved checkpoint. The project includes a streaming dataset pipeline that feeds training data from disk or network in real-time without loading the entire dataset into memory.

## Tags

### Artificial Intelligence & ML

- [Linear Attention Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/attention-mechanisms/linear-complexity-attention/linear-attention-training-frameworks.md) — Provides a complete training framework for sequence models using linear attention and state space mechanisms with hardware-optimized GPU kernels.
- [Autoregressive Inference Engines](https://awesome-repositories.com/f/artificial-intelligence-ml/autoregressive-inference-engines.md) — Ships a causal inference engine that generates text token-by-token using cached hidden states for efficient autoregressive decoding.
- [Distributed Model Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-model-checkpointing.md) — Ships a distributed checkpoint manager that splits model weights across multiple files for parallel loading and saving in multi-node training.
- [Fused GPU Kernel Composition](https://awesome-repositories.com/f/artificial-intelligence-ml/gpu-kernel-implementations/kernel-composition-frameworks/fused-gpu-kernel-composition.md) — Provides fused CUDA kernels that combine multiple neural operations into single GPU kernels to reduce memory bandwidth and launch overhead.
- [Selective State Space Models](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/sequence-models/selective-state-space-models.md) — Processes sequences using recurrent state updates that capture long-range dependencies with linear complexity.
- [Hybrid Model Architectures](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/frameworks/model-construction/neural-network-layers/convolution-layers/layered-architectures/sequential-text-processing-pipelines/hybrid-model-architectures.md) — Builds large language models that interleave standard attention with linear attention and state space layers to balance efficiency and global context.
- [Hybrid Layer Compositions](https://awesome-repositories.com/f/artificial-intelligence-ml/model-composition-architectures/hybrid-layer-compositions.md) — Supports building hybrid sequence models that interleave standard attention with linear attention and state space layers.
- [Hybrid Architecture Builders](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/hybrid-transformer-training/hybrid-sequence-model-training/hybrid-architecture-builders.md) — Provides a system for building hybrid sequence models that interleave standard attention with linear attention and state space layers within a single model architecture.
- [Streaming Dataset Loaders](https://awesome-repositories.com/f/artificial-intelligence-ml/dataset-batch-loading/streaming-dataset-loaders.md) — Includes a streaming dataset pipeline that feeds training data from disk or network in real-time without loading the entire dataset into memory.
- [Kernel Dispatchers](https://awesome-repositories.com/f/artificial-intelligence-ml/gpu-kernel-implementations/kernel-dispatchers.md) — Implements hardware-aware kernel dispatch that selects optimized CUDA kernels at runtime based on GPU architecture and tensor shapes.
- [Multi-Node Training Scaling](https://awesome-repositories.com/f/artificial-intelligence-ml/gpu-model-deployments/multi-node-training-scaling.md) — Distributes training across multiple GPU nodes by setting environment variables for inter-node communication. ([source](https://github.com/fla-org/flash-linear-attention/blob/main/examples/training.md))
- [Hybrid State Space Toolkits](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/sequence-models/selective-state-space-models/hybrid-state-space-toolkits.md) — Provides a toolkit for constructing hybrid sequence models that combine standard attention with linear and state space layers.
- [Model Checkpoint Converters](https://awesome-repositories.com/f/artificial-intelligence-ml/model-checkpoint-converters.md) — Transforms Hugging Face format checkpoints into the distributed format needed for training. ([source](https://github.com/fla-org/flash-linear-attention/blob/main/examples/training.md))
- [Domain-Adaptive Continued Pretraining](https://awesome-repositories.com/f/artificial-intelligence-ml/model-pretraining-frameworks/domain-adaptive-continued-pretraining.md) — Supports continuing training from a pretrained checkpoint using fresh data. ([source](https://github.com/fla-org/flash-linear-attention/blob/main/examples/training.md))
- [Checkpoint Format Transpilations](https://awesome-repositories.com/f/artificial-intelligence-ml/model-weight-export-formats/checkpoint-format-transpilations.md) — Provides a weight format transpilation utility for converting between Hugging Face and distributed checkpoint formats.
- [Training Checkpointers](https://awesome-repositories.com/f/artificial-intelligence-ml/training-checkpointers.md) — Automatically continues training from the last saved checkpoint after an interruption. ([source](https://github.com/fla-org/flash-linear-attention/blob/main/examples/training.md))

### Data & Databases

- [Long-Context Sequence Processors](https://awesome-repositories.com/f/data-databases/text-processing-pipelines/long-context-sequence-processors.md) — Trains and deploys sequence models that process long contexts with reduced memory and compute overhead using linear attention and state space mechanisms.
- [Model Weight Conversions](https://awesome-repositories.com/f/data-databases/vector-data-formats/format-conversion-utilities/model-weight-conversions.md) — Transforms model checkpoints between Hugging Face format and distributed training formats by remapping tensor layouts and metadata.

### DevOps & Infrastructure

- [Token Mixing Accelerators](https://awesome-repositories.com/f/devops-infrastructure/gpu-acceleration-libraries/token-mixing-accelerators.md) — Provides hardware-optimized token mixing layers and fused CUDA kernels that minimize memory bandwidth and launch overhead across different GPU architectures.

### Operating Systems & Systems Programming

- [Fused Token Mixing Kernels](https://awesome-repositories.com/f/operating-systems-systems-programming/gpu-memory-optimizations/fused-token-mixing-kernels.md) — Runs fused GPU kernels for token mixing operations that minimize memory bandwidth and launch overhead across different GPU architectures.

### Part of an Awesome List

- [From-Scratch Training](https://awesome-repositories.com/f/awesome-lists/ai/pre-trained-models/from-scratch-training.md) — Trains a new sequence model from scratch using a configurable script with optimizers and schedulers. ([source](https://github.com/fla-org/flash-linear-attention/blob/main/examples/training.md))