NVIDIA/apex
Apex
Apex is a high-performance toolkit for PyTorch designed to coordinate distributed training, execute fused GPU kernels, manage mixed precision, and implement optimized distributed optimizers. It provides specialized tools for scaling model training across multiple GPUs and nodes to increase processing speed and throughput.
The library features high-performance implementations of Adam and LAMB optimizers to reduce synchronization overhead and memory bottlenecks. It utilizes fused CUDA kernels to combine neural network operations, reducing memory overhead and increasing execution speed.
The toolkit further covers mixed precision training and gradient scaling to save memory while maintaining numerical stability. It also includes accelerated implementations of normalization layers such as LayerNorm, RMSNorm, and BatchNorm to improve training convergence.
Features
- Distributed GPU Training - Provides tools for scaling PyTorch model training across multiple GPUs and nodes.
- Distributed Training - Serves as a toolkit for configuring data and model parallelism across multiple PyTorch devices.
- Fused GPU Kernel Composition - Combines multiple mathematical operations into single GPU kernels to reduce memory traffic and increase throughput.
- Mixed Precision Training - Uses a blend of floating-point formats during training to reduce memory and increase throughput.
- Mixed-Precision Computing - Implements execution across 16-bit and 32-bit floating point formats to balance memory usage and stability.
- Deep Learning Optimization - Optimizes deep learning training speed and memory efficiency via fused kernels and optimized normalization.
- High-Performance Optimizer Implementations - Provides high-performance Adam and LAMB implementations to reduce synchronization overhead during large-scale training.
- Optimizer Performance Optimizations - Reduces synchronization overhead and memory bottlenecks using fused and distributed versions of Adam and LAMB.
- PyTorch Bindings - Provides C++ and CUDA extensions that bind high-performance operations to the PyTorch framework.
- Accelerated Normalization Layers - Provides accelerated implementations of LayerNorm, RMSNorm, and BatchNorm to improve training convergence speed.
- Distributed Training Sharding - Implements strategies for partitioning optimizer states across multiple GPUs to reduce memory footprint.
- Fused Neural Modules - Merges weight updates and gradient applications into single GPU passes to eliminate redundant memory reads.
- Distributed Optimizer Scaling - Manages memory overhead and synchronization for massive networks using distributed Adam and LAMB optimizers.
- Training Throughput Optimization - Increases processing speed and training throughput by spreading workloads across multiple GPUs and nodes.
- Loss Scaling Techniques - Uses loss scaling to prevent numerical underflow when training with lower precision floating point formats.
- Distributed Training Coordination - Coordinates and synchronizes machine learning training tasks across distributed GPU clusters.
- Developer Tools - Experimental extensions for performance.
