facebookresearch/lingua
Lingua
Lingua is a research framework for developing, training, and experimenting with large language model architectures and data strategies. It provides a lean codebase designed to facilitate the iteration of new model designs through a combination of distributed training orchestration and evaluation pipelines.
The framework includes a distributed training orchestrator that generates submission scripts and manages configurations for launching tasks across compute clusters. It utilizes a configuration management system that allows model parameters to be overridden via data classes and command-line arguments without modifying the source code.
The system further supports model development through checkpoint-based evaluation pipelines that run periodic scripts to track performance. Its architectural surface covers distributed runtime management, hierarchical configuration overrides, and modular templates for swapping model components.
Features
- LLM Architecture Research Frameworks - Provides a lean research framework for developing, training, and experimenting with new LLM architectures and data strategies.
- Distributed Training Orchestrators - Orchestrates the parallelization of model training across multiple processors and compute nodes.
- Distributed Training Runtimes - Orchestrates model parallelism and data distribution across multiple GPUs to scale large language model training.
- LLM Development Frameworks - Offers a comprehensive lean framework for the end-to-end research, training, and experimentation of LLMs.
- LLM Training Orchestrators - Manages the distributed orchestration of large-scale training and execution for language models across compute clusters.
- Periodic In-Training Evaluation - Provides a pipeline for running periodic evaluations on model saves to monitor performance during the training process.
- Large Language Model Training Frameworks - Provides a specialized framework for training and evaluating transformer-based models in multi-GPU environments.
- Model Checkpointing - Implements systems for saving and running evaluation scripts on neural network checkpoints during training.
- Modular Architectures - Uses a lean codebase structure with modular templates that allow rapid swapping of model components and data strategies.
- Training Script Generators - Automatically generates executable submission scripts for launching distributed training jobs across compute clusters.
- Command-Line Configurations - Enables tuning of model parameters and training settings via external files and command-line overrides.
- Job Submission Pipelines - Provides utilities for generating submission scripts and backing up configurations for cluster-managed training tasks.
- Hyperparameter Configurations - Implements a system to load hyperparameters from structured Python classes and command-line arguments.
- Hierarchical Configuration Overrides - Implements a priority-based hierarchy for loading settings from data classes, files, and command-line overrides.
- Fine-Tuning Frameworks - Lean and efficient codebase for model research.
- Model Training Frameworks - Lean and efficient codebase for LLM research.
- Fine-Tuning Frameworks - Lean codebase for model research and training.
