pytorch/torchtitan
Torchtitan
Torchtitan is a reference implementation for distributed deep learning built within the PyTorch ecosystem. It provides a framework for training large neural network models across multiple GPUs and nodes by combining several parallelism techniques, including fully sharded data parallelism (FSDP), tensor parallelism, and pipeline parallelism, making it possible to train models that exceed the memory capacity of a single device.
The system distinguishes itself through asynchronous checkpointing, which saves model and optimizer state to persistent storage without pausing the training loop, enabling fault tolerance and iterative experimentation. A unified composable parallelism scheduler allows data, tensor, and pipeline parallelism to be orchestrated from a single configuration, while a real-time monitoring tool logs loss, throughput, memory, and other metrics during training runs. The checkpoint format is designed to be directly loadable into conversion tools for subsequent fine‑tuning.
Additional capabilities include memory profile–driven autotuning that recommends optimal parallelism configurations, an elastic training coordinator that manages dynamic membership changes in the worker pool, and pipeline execution scheduling that minimises bubble time. These components collectively support large-scale distributed training with both high efficiency and operational flexibility.
Features
- Large-Scale Training Frameworks - Provides a reference implementation for training large models across multiple GPUs and nodes.
- Distributed Training - Combines data, tensor, and pipeline parallelism for distributed training across many GPUs.
- Fully Sharded Data Parallelism - Implements Fully Sharded Data Parallelism to distribute model states across devices.
- Model Parallelism - Distributes model layers and parameters across many GPUs using multiple parallelism strategies.
- Parallelism Orchestration - Orchestrates data, tensor, and pipeline parallelism from a single configuration.
- Training Pipeline Schedules - Schedules micro-batches across pipeline stages using the one-forward-one-back algorithm.
- PyTorch Training Frameworks - Serves as a reference implementation for distributed PyTorch training with multiple parallelism strategies.
- Asynchronous Checkpoint Managers - Provides asynchronous checkpointing for interruption‑resilient PyTorch training.
- Tensor Parallelism - Splits large layers across devices using row and column sharding techniques.
- Training Checkpointing - Saves and restores model state asynchronously without pausing the training loop.
- Asynchronous Training Persisters - Saves model and optimizer state asynchronously without pausing the training loop.
- Performance Monitoring - Logs loss, throughput, memory, and other metrics in real time during PyTorch training runs.
- Training Progress Monitors - Logs loss, memory, and throughput metrics for real-time training progress tracking.
- Checkpoint Saving and Restoration - Saves model state asynchronously and loads it for fine-tuning in a compatible format.
- Memory-Driven Autotuning - Analyzes memory and communication costs to autotune parallelism configurations.
- Elastic Training Scaling - Manages dynamic addition and removal of worker nodes during training without restart.
- Model Training Frameworks - Native PyTorch library for large-scale model training.
