mosaicml/llm-foundry

Features

• Distributed Training - Provides a high-performance framework for scaling LLM training across multiple nodes and GPUs using parameter sharding.
• Cloud Dataset Streaming - Implements a pipeline for compressing and streaming large-scale datasets from cloud storage during training.
• Cloud-Native Data Streaming - Streams training datasets directly from cloud object storage to bypass local disk capacity limits.
• Distributed ML Trainers - Ships a scalable trainer for distributing LLM workloads across compute clusters using parameter sharding.
• Distributed Training Sharding - Partitions model parameters and optimizer states across multiple GPU nodes to scale large-scale training throughput.
• Foundation Model Pre-training - Enables the training of base language models from scratch on massive corpora using configuration-driven pipelines.
• Multi-Node Training Scaling - Distributes large-scale training workloads across multiple GPU nodes using parameter sharding to maintain high throughput.
• Large Language Model Fine-Tuning - Supports specializing pre-trained models via supervised instruction tuning and parameter-efficient adaptation.
• Low-Rank Adaptation - Implements low-rank adaptation (LoRA) to reduce memory and compute costs during model fine-tuning.
• Large Language Model Training Frameworks - Provides a complete framework for foundation model pre-training and supervised fine-tuning of large language models.
• Low-Rank Adaptation - Implements parameter-efficient fine-tuning using low-rank adaptation to reduce computational costs.
• Supervised Fine-Tuning - Adapts pre-trained models to specific tasks or datasets using supervised or instruction-based training methodologies.
• Streaming Dataset Loaders - Loads remote datasets in a compressed streaming format to eliminate local disk bottlenecks.
• Mixed Precision Training - Employs eight-bit mixed-precision training to accelerate compute speed and reduce memory usage on compatible hardware.
• FP8 Training Implementations - Uses eight-bit floating point precision and specialized layers to accelerate training speed on compatible hardware.
• Model Performance Benchmarking - Measures model accuracy and quality using standardized benchmarks and in-context learning tests.
• Model Performance Benchmarking - Verifies model convergence by measuring performance against standard in-context-learning benchmarks using consistent prompt sets.
• LLM Performance Evaluators - Quantifies model accuracy and reliability by running inference on specific task subsets and comparing predictions against ground truth.
• Training Dataset Preparation - Prepares and compresses raw data into streaming formats optimized for loading from cloud storage.
• Data Format Transformations - Transforms raw data into compressed, streaming-compatible formats to improve training efficiency and throughput.
• Configuration-Driven Pipelines - Implements training workflows controlled by external configuration files to ensure reproducibility across datasets and hardware.
• LLM Evaluation - Provides tools for verifying model convergence and quality using standard in-context-learning benchmarks.
• Fine-Tuning Frameworks - Training code for foundation model development.
• Large Language Models - Tooling for training and fine-tuning large-scale models.
• Fine-Tuning Frameworks - Training code for foundation model development.
• Large Language Models (LLMs) - Listed in the “Large Language Models (LLMs)” section of the The Incredible Pytorch awesome list.

Star-Verlauf