uber/horovod
Horovod
Horovod is a distributed deep learning framework designed to scale machine learning training across multiple GPUs and nodes. It functions as an orchestrator for multi-GPU scaling and a tool for distributed gradient averaging, allowing users to increase compute capacity without rewriting core model logic.
The project provides a consistent communication interface that supports multi-framework model distribution across TensorFlow, PyTorch, Keras, and MXNet. It leverages an MPI distributed training library to synchronize gradients across processes using collective communication operations.
The system covers gradient synchronization, large-scale model distribution, and performance tuning. It includes capabilities for overlapping communication with computation, batching tensor-based updates, and recording activity timelines to analyze distributed performance and identify bottlenecks.
Features
- Distributed Deep Learning - Scales deep learning model training across multiple compute nodes and GPUs using collective communication.
- Communication-Computation Overlap - Implements techniques to hide network latency by overlapping gradient synchronization with active computation.
- Distributed Deep Learning Frameworks - Acts as a unified platform for scaling machine learning training across multiple GPUs and nodes.
- Distributed GPU Computing - Expands compute capacity by distributing training scripts across multiple GPU hosts.
- Distributed Training Frameworks - Provides a consistent communication interface for distributed training across TensorFlow, PyTorch, Keras, and MXNet.
- Communication Optimization - Implements communication batching and computation-communication overlapping to reduce overhead and improve scaling efficiency.
- Distributed Gradient Synchronization - Calculates the average of gradients across distributed processes to synchronize model weights.
- Large Scale Training - Distributes machine learning workloads across multiple GPUs or nodes to accelerate the processing of large datasets.
- Multi-GPU Training Utilities - Distributes training scripts across multiple GPUs or hosts to increase compute capacity without rewriting core logic.
- Ring-Allreduce Algorithms - Uses a ring-allreduce algorithm to synchronize gradients across nodes while minimizing network congestion.
- MPI Communication - Leverages MPI and collective operations to synchronize gradients across distributed deep learning processes.
- Initial State Broadcasting - Synchronizes initial model weights from a root process to all worker nodes to ensure identical starting parameters.
- Training Performance Profiling - Provides tools to record precise timestamps for communication and computation to identify training bottlenecks.
- Performance Tuning - Optimizes communication overhead and identifies bottlenecks to maximize throughput in multi-node environments.
- Tensor Communication Batching - Groups multiple small gradient updates into a single large buffer to reduce network communication frequency.
- Framework Call Interceptors - Provides a wrapper that intercepts deep learning framework calls to implement distributed communication across a cluster.
- Distributed State Synchronizers - Synchronizes initial variable states from a primary worker to all other processes for consistent initialization.
- Distributed Training Timelines - Provides timeline-based recording of internal activity to identify synchronization bottlenecks and measure distributed training efficiency.
- Distributed Computing - Enables distributed training across multiple deep learning frameworks.
