# mosaicml/composer

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/mosaicml-composer).**

5,485 stars · 464 forks · Python · Apache-2.0

## Links

- GitHub: https://github.com/mosaicml/composer
- Homepage: http://docs.mosaicml.com
- awesome-repositories: https://awesome-repositories.com/repository/mosaicml-composer.md

## Description

Composer is a PyTorch distributed training framework designed for scaling large-scale models across multi-node GPU clusters. It functions as a large language model trainer, a distributed model optimizer, and a training lifecycle manager.

The project differentiates itself as a deep learning regularization library, providing specialized optimization techniques such as Sharpness Aware Minimization, MixUp, and CutMix to improve model generalization. It further distinguishes its training flow through the use of sequence length warmup, progressive layer freezing, and sharded-state checkpointing for large-scale model recovery.

The framework covers a broad capability surface including distributed training orchestration, mixed-precision hardware management, and cloud-native data streaming. It also provides extensive monitoring and observability tools for GPU memory diagnostics, training divergence detection, and throughput tracking.

The project includes a command-line launcher to automate the execution of multi-GPU training jobs across nodes.

## Tags

### Artificial Intelligence & ML

- [Language Model Trainers](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-trainers.md) — Provides a comprehensive system for training LLMs with sharded checkpoints, sequence length warmup, and distributed data parallelism.
- [Large-Scale Training Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/large-scale-training-frameworks.md) — Implements a framework for scaling large-scale model training across multi-node GPU clusters using PyTorch.
- [Dynamic Batch Size Adjustment](https://awesome-repositories.com/f/artificial-intelligence-ml/batch-size-tuning/automatic-batch-size-optimization/dynamic-batch-size-adjustment.md) — Adjusts microbatch sizes and gradient accumulation rates dynamically to prevent out-of-memory errors. ([source](https://docs.mosaicml.com/projects/composer/en/stable/examples/auto_microbatching.html))
- [Deep Learning Regularization Libraries](https://awesome-repositories.com/f/artificial-intelligence-ml/deep-learning-regularization-libraries.md) — Provides a collection of training algorithms including MixUp and Sharpness Aware Minimization to improve generalization.
- [Data-Parallel Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks/data-parallel-training.md) — Synchronizes training workloads across multi-node clusters by managing data samplers and global batch sizes.
- [Distributed Training](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-frameworks/distributed-training.md) — Executes model training with configurable hardware selection, mixed-precision acceleration, and reproducible seeding. ([source](https://docs.mosaicml.com/projects/composer/en/stable/examples/finetune_huggingface.html))
- [Distributed Training Metadata](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-metadata.md) — Provides access to global rank and world size to implement rank-specific logic across multi-node GPU clusters. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html))
- [Distributed Training Optimizers](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-optimizers.md) — Improves training efficiency through mixed-precision, gradient accumulation, and communication-efficient optimization. ([source](https://docs.mosaicml.com/projects/composer/en/stable/examples/migrate_from_ptl.html))
- [Distributed Training Orchestration](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-orchestration.md) — Orchestrates training across multi-node clusters by abstracting parallelism and distributed data loading. ([source](https://docs.mosaicml.com/projects/composer/en/stable/))
- [Gradient Clipping Utilities](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/gradient-clipping-utilities.md) — Limits gradient values or norms to prevent unstable updates and exploding gradients. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.GradientClipping.html))
- [Hardware Acceleration Backends](https://awesome-repositories.com/f/artificial-intelligence-ml/hardware-acceleration-backends.md) — Directs model execution across various hardware backends including CPUs, GPUs, and TPUs. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.devices.html))
- [Standardized Training Workflows](https://awesome-repositories.com/f/artificial-intelligence-ml/language-model-trainers/multimodal-training-interfaces/multimodal-model-trainers/standardized-training-workflows.md) — Implements consistent loops and interfaces for forward passes, loss computation, and evaluation across different model architectures. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.ComposerModel.html))
- [Training Lifecycle Management](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/pipelines-and-orchestration/training-orchestration-systems/training-lifecycle-management.md) — Provides a comprehensive manager for handling model checkpoints, automated training stops, and cloud storage synchronization.
- [Progressive](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/utilities/layer-freezing/progressive.md) — Freezes network layers incrementally during training to optimize stability. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.LayerFreezing.html))
- [Model Parallelism](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/training-frameworks/model-training-pipelines/model-parallelism.md) — Distributes large models across multiple GPUs or nodes using data, shard, or tensor parallelism. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))
- [Progressive Sequence Length Warmup](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-frameworks/vision-model-training/hybrid-transformer-training/hybrid-sequence-model-training/user-behavior-sequence-training/variable-length-sequence-training/progressive-sequence-length-warmup.md) — Increases input sequence length progressively during early training to improve stability. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.SeqLengthWarmup.html))
- [Progressive Input Scaling](https://awesome-repositories.com/f/artificial-intelligence-ml/reinforcement-learning-training-pipelines/rl-training-workflows/progressive-context-length-scaling/progressive-input-scaling.md) — Gradually increases input resolution or sequence length during training to accelerate convergence. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.ProgressiveResizing.html))
- [Sharpness-Aware Minimization](https://awesome-repositories.com/f/artificial-intelligence-ml/sharpness-aware-minimization.md) — Minimizes the sharpness of the loss landscape to improve model generalization and robustness. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.SAM.html))
- [Tensor Parallelism](https://awesome-repositories.com/f/artificial-intelligence-ml/tensor-parallelism.md) — Shards individual tensors across multiple devices according to a layer plan to train large models. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html))
- [Training Checkpoint Persistence](https://awesome-repositories.com/f/artificial-intelligence-ml/training-checkpoint-persistence.md) — Persists training state or weights to storage at specified intervals using customizable naming. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.CheckpointSaver.html))
- [Automatic Failure Resumption](https://awesome-repositories.com/f/artificial-intelligence-ml/training-checkpointing/automatic-failure-resumption.md) — Automatically detects the latest saved checkpoint and restarts training after a failure to maintain progress. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))
- [Training Loop Control](https://awesome-repositories.com/f/artificial-intelligence-ml/training-loop-control.md) — Manages training flow using progressive layer freezing, sequence length warmup, and weight maintenance. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.algorithms.html))
- [Length Extrapolation Biases](https://awesome-repositories.com/f/artificial-intelligence-ml/attention-bias-generation/length-extrapolation-biases.md) — Biases attention matrices to favor nearby tokens to improve extrapolation to unseen sequence lengths. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.Alibi.html))
- [Checkpoint-Based Recovery](https://awesome-repositories.com/f/artificial-intelligence-ml/checkpoint-based-recovery.md) — Provides the ability to restore model states from the latest checkpoint to recover from training failures. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/resumption.html))
- [Cloud Dataset Streaming](https://awesome-repositories.com/f/artificial-intelligence-ml/cloud-dataset-streaming.md) — Downloads training data from cloud blob storage on the fly to avoid local disk bottlenecks. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))
- [Cloud-Native Data Streaming](https://awesome-repositories.com/f/artificial-intelligence-ml/cloud-native-data-streaming.md) — Downloads training datasets from remote object storage on the fly to bypass local disk capacity limits.
- [Distributed Data Coordination](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-data-coordination.md) — Ensures different devices receive unique data batches using compatible distributed samplers. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html))
- [Distributed Model Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-model-checkpointing.md) — Supports saving and loading large-scale model states in sharded formats across multiple compute nodes. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html))
- [Distributed Training Managers](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-managers.md) — Launches multi-GPU training jobs by setting environment variables and managing process execution across nodes. ([source](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html))
- [Generative Model Evaluation](https://awesome-repositories.com/f/artificial-intelligence-ml/generative-model-evaluation.md) — Produces periodic sample outputs from prompts to assess and monitor generative model progress. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))
- [Gradient Norm Monitors](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/gradient-norm-monitors.md) — Computes and logs gradient L2 norms to monitor training stability and weight updates. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.OptimizerMonitor.html))
- [Hardware Device Management](https://awesome-repositories.com/f/artificial-intelligence-ml/hardware-device-management.md) — Manages the placement of models, optimizers, and data batches across specific CPU and GPU hardware. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.devices.Device.html))
- [Training Parameter Averaging](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/architectures/instruction-tuned-language-models/weight-space-merging-techniques/weighted-parameter-averaging/training-parameter-averaging.md) — Averages model weights sampled near the end of training to improve final accuracy. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.SWA.html))
- [Gradient Flow Stabilizers](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/machine-learning-training/utilities/gradient-optimization-techniques/gradient-flow-stabilizers.md) — Clips gradients and manages layer freezing to stabilize and accelerate the training process. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.functional.html))
- [Experiment Tracking Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/training-monitoring-and-profiling/training-observability-systems/experiment-tracking-integrations.md) — Integrates training data and metrics with external platforms for automated experiment tracking and analysis. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))
- [Training Safety Monitors](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/training-monitoring-and-profiling/training-observability-systems/training-monitoring-tools/training-safety-monitors.md) — Monitors loss values and automatically detects NaN events to prevent unstable model convergence. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.NaNMonitor.html))
- [Model Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/model-checkpointing.md) — Implements systems for saving and restoring neural network states to preserve training progress. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))
- [Memory Layout Optimizers](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training-optimizers/memory-layout-optimizers.md) — Optimizes GPU utilization by setting the model memory format to channels-last. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.ChannelsLast.html))
- [Training Task Automation](https://awesome-repositories.com/f/artificial-intelligence-ml/model-training/early-stopping-callbacks/training-task-automation.md) — Halts training automatically when specific metrics reach a threshold or stop improving. ([source](https://docs.mosaicml.com/projects/composer/en/stable/trainer/callbacks.html))
- [Parameter Weight Smoothing](https://awesome-repositories.com/f/artificial-intelligence-ml/model-weight-reconstruction/weight-smoothing/exponential-moving-average-weight-updates/parameter-weight-smoothing.md) — Tracks a secondary set of model weights using an exponential moving average to improve evaluation. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.EMA.html))
- [Optimization Algorithm Injections](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-training-pipelines/training-execution-loops/optimization-algorithm-injections.md) — Integrates specialized optimization techniques and regularization methods directly into the core training execution flow.
- [Training Lifecycle Hooks](https://awesome-repositories.com/f/artificial-intelligence-ml/neural-training-pipelines/training-execution-loops/training-lifecycle-hooks.md) — Allows users to inject custom logic into the training loop by triggering functions at specific lifecycle events.
- [Stochastic Depth Regularization](https://awesome-repositories.com/f/artificial-intelligence-ml/overfitting-reduction-techniques/stochastic-depth-regularization.md) — Provides stochastic depth regularization to randomly drop network paths during training and prevent overfitting in deep models. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.StochasticDepth.html))
- [Mixed-Precision Orchestration](https://awesome-repositories.com/f/artificial-intelligence-ml/precision-quantization/mixed-precision-orchestration.md) — Manages tensor casting and memory formats across different accelerators to balance computational speed and numerical accuracy.
- [Trainer State Serialization](https://awesome-repositories.com/f/artificial-intelligence-ml/trainer-state-serialization.md) — Captures the internal trainer state into portable dictionaries to enable seamless resumption after hardware failures.
- [Gradient Accumulation Strategies](https://awesome-repositories.com/f/artificial-intelligence-ml/training-convergence-optimization/batch-size-scaling/gradient-accumulation-strategies.md) — Simulates larger batch sizes by accumulating gradients over multiple micro-batches to reduce memory overhead. ([source](https://docs.mosaicml.com/projects/composer/en/stable/examples/migrate_from_ptl.html))
- [Training Lifecycle Hooks](https://awesome-repositories.com/f/artificial-intelligence-ml/training-lifecycle-hooks.md) — Injects custom logic into the training loop by hooking into events like epoch starts or batch ends. ([source](https://docs.mosaicml.com/projects/composer/en/stable/trainer/callbacks.html))
- [Training Progress Monitors](https://awesome-repositories.com/f/artificial-intelligence-ml/training-progress-monitors.md) — Measures progress across epochs and batches to trigger specific behaviors at defined training intervals. ([source](https://docs.mosaicml.com/projects/composer/en/latest/trainer/time.html))

### Part of an Awesome List

- [Mixup Augmentations](https://awesome-repositories.com/f/awesome-lists/ai/data-augmentation/mixup-augmentations.md) — Creates combinations of training examples and targets to reduce generalization error. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.algorithms.MixUp.html))
- [Standardized Evaluation Harnesses](https://awesome-repositories.com/f/awesome-lists/ai/model-evaluation-and-benchmarking/standardized-evaluation-harnesses.md) — Converts diverse data sources into consistent evaluator objects to ensure standardized and reproducible model benchmarking. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.core.ensure_evaluator.html))
- [Computation and Optimization](https://awesome-repositories.com/f/awesome-lists/devtools/computation-and-optimization.md) — PyTorch library for faster neural network training with higher accuracy.
- [Machine Learning Frameworks](https://awesome-repositories.com/f/awesome-lists/devtools/machine-learning-frameworks.md) — Library for efficient and scalable model training.

### Data & Databases

- [Sharded Checkpoint Storage](https://awesome-repositories.com/f/data-databases/data-checkpointing/checkpoint-metadata-tracking/sharded-checkpoint-storage.md) — Saves and restores model weights as distributed shards to support large-scale models across varying GPU counts.
- [Elastic Resumption](https://awesome-repositories.com/f/data-databases/data-checkpointing/checkpoint-metadata-tracking/sharded-checkpoint-storage/elastic-resumption.md) — Allows resuming sharded model states across different hardware configurations and GPU counts. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))
- [Training Memory Optimizers](https://awesome-repositories.com/f/data-databases/memory-optimization-strategies/training-memory-optimizers.md) — Lowers the peak memory footprint by freeing training metric memory immediately after loss calculation. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.FreeOutputs.html))
- [NaN Loss Training Halts](https://awesome-repositories.com/f/data-databases/numeric-calculators/numeric-extremes-calculations/nan-aware-extremes-calculators/nan-verifications/nan-loss-training-halts.md) — Detects NaN values in loss computations to immediately halt training and prevent corrupted model weights. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))
- [Cloud Synchronization](https://awesome-repositories.com/f/data-databases/state-checkpointing/model-checkpoints/cloud-synchronization.md) — Enables storing and loading model checkpoints directly from remote cloud buckets or object storage. ([source](https://cdn.jsdelivr.net/gh/mosaicml/composer@main/README.md))

### Development Tools & Productivity

- [GPU Device Assignment](https://awesome-repositories.com/f/development-tools-productivity/diagramming-tools/process-and-flow-mapping/process-configuration-mappings/process-to-socket-mappings/gpu-device-assignment.md) — Manages the selection of GPU devices for training and configures TF32 matrix multiplications. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.devices.DeviceGPU.html))

### DevOps & Infrastructure

- [Model Export Formats](https://awesome-repositories.com/f/devops-infrastructure/deployment-management/model-export-formats.md) — Converts trained models into portable formats optimized for production deployment and storage. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))

### Hardware & IoT

- [Tensor Layout Optimizations](https://awesome-repositories.com/f/hardware-iot/tensor-layout-optimizations.md) — Specifies the order of tensor dimensions in memory to improve hardware processing unit utilization. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.core.MemoryFormat.html))

### Scientific & Mathematical Computing

- [Floating-Point Precision Conversions](https://awesome-repositories.com/f/scientific-mathematical-computing/floating-point-precision-conversions.md) — Casts tensor operations to specific floating-point precisions to balance computational speed and numerical accuracy. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.core.get_precision_context.html))

### System Administration & Monitoring

- [Throughput and ETA Monitors](https://awesome-repositories.com/f/system-administration-monitoring/activity-monitors/activity-progress-monitors/task-progress-monitors/throughput-and-eta-monitors.md) — Tracks real-time processing speed in terms of batches, samples, and tokens per second. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.SpeedMonitor.html))
- [Training Progress Recording](https://awesome-repositories.com/f/system-administration-monitoring/global-metric-registries/custom-metric-collectors/custom-metric-recording/training-progress-recording.md) — Captures training metrics and checkpoints by routing them to local files, consoles, and monitoring platforms. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.loggers.html))
- [Hardware Monitoring Utilities](https://awesome-repositories.com/f/system-administration-monitoring/hardware-monitoring-utilities.md) — Tracks GPU and CPU metrics including occupancy, temperature, and power consumption during training. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.SystemMetricsMonitor.html))
- [Memory Usage Analyzers](https://awesome-repositories.com/f/system-administration-monitoring/memory-usage-analyzers.md) — Captures memory snapshots and visualizes allocation states to identify the causes of memory errors. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))
- [Memory Allocation Visualizers](https://awesome-repositories.com/f/system-administration-monitoring/memory-usage-analyzers/memory-allocation-debuggers/memory-allocation-visualizers.md) — Generates memory snapshots and flamegraphs to visualize and diagnose GPU memory allocation issues. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.OOMObserver.html))
- [GPU Memory Monitors](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/system-usage-monitoring/system-usage-monitors/gpu-memory-monitors.md) — Logs CUDA memory statistics during training batches to track VRAM utilization and detect memory leaks. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.MemoryMonitor.html))
- [Memory Snapshotting](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/system-usage-monitoring/system-usage-monitors/gpu-memory-monitors/memory-snapshotting.md) — Records tensor memory allocations and out-of-memory events to facilitate GPU debugging. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.MemorySnapshot.html))
- [Training Metric Monitors](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/observability-platforms/metric-performance-monitors/training-metric-monitors.md) — Provides tools to track machine learning performance indicators like throughput and memory usage during training. ([source](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/composer.callbacks.html))