pytorch/torchtitan

Features

• Large-Scale Training Frameworks - Provides a reference implementation for training large models across multiple GPUs and nodes.
• Distributed Training - Combines data, tensor, and pipeline parallelism for distributed training across many GPUs.
• Fully Sharded Data Parallelism - Implements Fully Sharded Data Parallelism to distribute model states across devices.
• Model Parallelism - Distributes model layers and parameters across many GPUs using multiple parallelism strategies.
• Parallelism Orchestration - Orchestrates data, tensor, and pipeline parallelism from a single configuration.
• Training Pipeline Schedules - Schedules micro-batches across pipeline stages using the one-forward-one-back algorithm.
• PyTorch Training Frameworks - Serves as a reference implementation for distributed PyTorch training with multiple parallelism strategies.
• Asynchronous Checkpoint Managers - Provides asynchronous checkpointing for interruption‑resilient PyTorch training.
• Tensor Parallelism - Splits large layers across devices using row and column sharding techniques.
• Training Checkpointing - Saves and restores model state asynchronously without pausing the training loop.
• Asynchronous Training Persisters - Saves model and optimizer state asynchronously without pausing the training loop.
• Performance Monitoring - Logs loss, throughput, memory, and other metrics in real time during PyTorch training runs.
• Training Progress Monitors - Logs loss, memory, and throughput metrics for real-time training progress tracking.
• Checkpoint Saving and Restoration - Saves model state asynchronously and loads it for fine-tuning in a compatible format.
• Memory-Driven Autotuning - Analyzes memory and communication costs to autotune parallelism configurations.
• Elastic Training Scaling - Manages dynamic addition and removal of worker nodes during training without restart.
• Model Training Frameworks - Native PyTorch library for large-scale model training.

Star history