Lightning-AI/lit-llama
Lit Llama
Lit-llama es un framework de implementación basado en PyTorch para el modelo de lenguaje LLaMA, que proporciona un sistema para pre-entrenamiento, ajuste fino e inferencia de alto rendimiento. Incluye un pipeline de pre-entrenamiento para crear modelos de lenguaje fundamentales desde cero y herramientas para ejecutar pesos pre-entrenados para generar texto natural y predecir secuencias.
El proyecto proporciona toolkits especializados para el ajuste fino eficiente en parámetros utilizando adaptación de bajo rango (LoRA) y adaptadores ligeros. También incluye una librería de cuantización que reduce las huellas de memoria del modelo a través de precisión de cuatro y ocho bits para permitir la ejecución en hardware con recursos limitados.
El framework incorpora un diseño de transformador simplificado y emplea flash attention para optimizar la memoria y la velocidad. Además, gestiona datasets a gran escala a través de formatos de datos en streaming para evitar cargar corpora completos en la memoria del sistema.
Features
- LLM Development Frameworks - Provides a complete PyTorch-based pipeline for building and refining LLaMA-based models from scratch to deployment.
- Foundation Model Pre-training - Includes a pre-training pipeline for creating foundational language models from scratch using large-scale datasets.
- Large-Scale Model Training - Provides a complete pipeline for training foundational language models from scratch using optimized data streaming.
- Llama Model Inference - Provides runtimes and pipelines specifically optimized for executing Llama family model weights.
- LLM Implementations - Provides a complete PyTorch-based implementation of the LLaMA language model for training and inference.
- Low-Rank Adaptation - Implements low-rank adaptation to reduce trainable parameters during the fine-tuning process.
- Parameter Efficient Fine-Tuning - Implements memory-efficient adaptation techniques like LoRA and lightweight adapters to specialize models on custom datasets.
- Pre-training Pipelines - Provides the data processing and modeling code required for the initial large-scale training phase.
- From-Scratch Training - Enables the training of foundational models from scratch on raw datasets.
- Flash-Attention Implementations - Implements optimized attention mechanisms that reduce memory access and compute overhead for long sequences.
- LoRA Fine-Tuning Tools - Offers a toolkit for parameter-efficient fine-tuning of models using low-rank adaptation.
- Minimalist Transformer Implementations - Utilizes a simplified transformer design to ensure high training efficiency and minimal computational overhead.
- 4-Bit Compressors - Includes tools to reduce model weight memory usage through 4-bit post-training quantization.
- Weight Adapters - Provides support for using small weight adapters to specialize pretrained models for specific tasks.
- Model Quantization - Reduces model precision using Int8 or 4-bit formats to optimize memory and performance.
- Weight Quantization - Compresses floating point weights into eight-bit integers to reduce the model memory footprint.
- Adapter-Based Specializations - Refines pretrained models on custom datasets using lightweight adapters to improve domain-specific accuracy.
- Memory-Efficient Data Streaming - Processes massive datasets in small chunks from disk to prevent system memory overload during pre-training.
- GPTQ Quantization Libraries - Ships a quantization library that reduces memory footprints via GPTQ-based 4-bit and 8-bit precision.
- Language Model Frameworks - Provides a lightweight implementation of the LLaMA architecture.
- LLM Frameworks - Independent, Apache 2.0 licensed implementation of LLaMA.
- Natural Language Processing - Listed in the “Natural Language Processing” section of the FunNLP awesome list.
- Open Source Models - Implements efficient fine-tuning and training for language models.
