Lightning-AI/lit-llama
Lit Llama
Lit-llama is a PyTorch-based implementation framework for the LLaMA language model, providing a system for pre-training, fine-tuning, and high-performance inference. It includes a pre-training pipeline for creating foundational language models from scratch and tools for running pretrained weights to generate natural text and predict sequences.
The project provides specialized toolkits for parameter-efficient fine-tuning using low-rank adaptation and lightweight adapters. It also includes a quantization library that reduces model memory footprints through four-bit and eight-bit precision to enable execution on hardware with limited resources.
The framework incorporates a simplified transformer design and employs flash attention to optimize memory and speed. It further manages large-scale datasets through streaming data formats to avoid loading entire corpora into system memory.
Features
- LLM Development Frameworks - Provides a complete PyTorch-based pipeline for building and refining LLaMA-based models from scratch to deployment.
- Foundation Model Pre-training - Includes a pre-training pipeline for creating foundational language models from scratch using large-scale datasets.
- Large-Scale Model Training - Provides a complete pipeline for training foundational language models from scratch using optimized data streaming.
- Llama Model Inference - Provides runtimes and pipelines specifically optimized for executing Llama family model weights.
- LLM Implementations - Provides a complete PyTorch-based implementation of the LLaMA language model for training and inference.
- Low-Rank Adaptation - Implements low-rank adaptation to reduce trainable parameters during the fine-tuning process.
- Parameter Efficient Fine-Tuning - Implements memory-efficient adaptation techniques like LoRA and lightweight adapters to specialize models on custom datasets.
- Pre-training Pipelines - Provides the data processing and modeling code required for the initial large-scale training phase.
- From-Scratch Training - Enables the training of foundational models from scratch on raw datasets.
- Flash-Attention Implementations - Implements optimized attention mechanisms that reduce memory access and compute overhead for long sequences.
- LoRA Fine-Tuning Tools - Offers a toolkit for parameter-efficient fine-tuning of models using low-rank adaptation.
- Minimalist Transformer Implementations - Utilizes a simplified transformer design to ensure high training efficiency and minimal computational overhead.
- 4-Bit Compressors - Includes tools to reduce model weight memory usage through 4-bit post-training quantization.
- Weight Adapters - Provides support for using small weight adapters to specialize pretrained models for specific tasks.
- Model Quantization - Reduces model precision using Int8 or 4-bit formats to optimize memory and performance.
- Weight Quantization - Compresses floating point weights into eight-bit integers to reduce the model memory footprint.
- Adapter-Based Specializations - Refines pretrained models on custom datasets using lightweight adapters to improve domain-specific accuracy.
- Memory-Efficient Data Streaming - Processes massive datasets in small chunks from disk to prevent system memory overload during pre-training.
- GPTQ Quantization Libraries - Ships a quantization library that reduces memory footprints via GPTQ-based 4-bit and 8-bit precision.
- Language Model Frameworks - Provides a lightweight implementation of the LLaMA architecture.
- LLM Frameworks - Independent, Apache 2.0 licensed implementation of LLaMA.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
- Open Source Models - Implements efficient fine-tuning and training for language models.
