# horovod/horovod

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [awesome-repositories.com](https://awesome-repositories.com/repository/horovod-horovod).**

14,686 stars · 2,244 forks · Python · NOASSERTION

## Links

- GitHub: https://github.com/horovod/horovod
- Homepage: http://horovod.ai
- awesome-repositories: https://awesome-repositories.com/repository/horovod-horovod.md

## Description

Horovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models.

The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batching and half-precision gradient compression to reduce network bandwidth requirements.

The framework covers a broad set of capabilities including cluster orchestration across Kubernetes, Spark, and Ray, as well as hardware-aware resource mapping for CPUs and GPUs. It provides tools for distributed data management, such as parallel loading from Parquet files and offloaded preprocessing. Performance is further supported by RDMA network optimization, execution tracing, and Bayesian training optimization to maximize throughput.

Deployment is supported through containerized training images and orchestrated environments for high-performance compute clusters.

## Tags

### Artificial Intelligence & ML

- [Distributed Deep Learning Frameworks](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-deep-learning-frameworks.md) — Functions as a unified platform for scaling deep learning model training across multiple GPUs and compute nodes.
- [Distributed Gradient Synchronization](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-computation/distributed-gradient-synchronization.md) — Implements all-reduce collective communication to synchronize gradients across distributed workers.
- [Communication Layers](https://awesome-repositories.com/f/artificial-intelligence-ml/communication-layers.md) — Provides a communication abstraction layer that decouples the distributed training logic from specific network hardware and libraries.
- [Distributed Deep Learning](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-deep-learning.md) — Scales the training of deep learning models across multiple GPUs and compute nodes to accelerate convergence.
- [Initial State Broadcasting](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-model-state-management/initial-state-broadcasting.md) — Broadcasts initial variable states from a lead process to all workers to ensure consistent model weight initialization. ([source](https://github.com/horovod/horovod/blob/master/docs/pytorch.rst))
- [Distributed Training Orchestration](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-orchestration.md) — Provides orchestration for distributed training jobs specifically integrated with Apache Spark clusters. ([source](https://github.com/horovod/horovod/blob/master/docs/install.rst))
- [Distributed Training Orchestrators](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training-orchestrators.md) — Coordinates process ranks and resource allocation to parallelize model training across multiple processors and nodes.
- [Communication Optimization](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-training/communication-optimization.md) — Optimizes collective communication using all-reduce and all-gather operations to synchronize gradients efficiently.
- [Distributed Training Rate Scaling](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/model-fine-tuning-adaptation/learning-rate-schedulers/distributed-training-rate-scaling.md) — Adjusts the learning rate based on the number of active workers to compensate for increased global batch sizes. ([source](https://github.com/horovod/horovod/blob/master/docs/pytorch.rst))
- [Elastic Scaling](https://awesome-repositories.com/f/artificial-intelligence-ml/training-engines/elastic-scaling.md) — Provides a dynamic elastic training engine that adjusts worker counts at runtime and recovers from host failures.
- [Spark Integrations](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-deep-learning/spark-integrations.md) — Enables executing distributed training jobs on Apache Spark clusters to leverage existing resource management.
- [Distributed Model Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/distributed-model-checkpointing.md) — Manages the persistence of model states across distributed nodes while preventing concurrent write corruption. ([source](https://github.com/horovod/horovod/blob/master/docs/tensorflow.rst))
- [Gradient Compression Techniques](https://awesome-repositories.com/f/artificial-intelligence-ml/gradient-compression-techniques.md) — Reduces network bandwidth by compressing tensor gradients using half-precision formats during synchronization.
- [Offloaded Preprocessing](https://awesome-repositories.com/f/artificial-intelligence-ml/machine-learning/infrastructure/model-training-and-tuning/data-and-checkpointing/data-preprocessing/offloaded-preprocessing.md) — Offloads CPU-intensive dataset transformations to a dedicated cluster of processes to prevent data bottlenecks during GPU training. ([source](https://github.com/horovod/horovod/blob/master/docs/tensorflow.rst))
- [Worker-Specific Serialization](https://awesome-repositories.com/f/artificial-intelligence-ml/model-checkpointing/worker-specific-serialization.md) — Prevents filesystem corruption by restricting model checkpoint saving to a single designated worker.
- [Performance Tuning](https://awesome-repositories.com/f/artificial-intelligence-ml/performance-tuning.md) — Provides methods for optimizing computational throughput and network bandwidth in distributed machine learning pipelines.
- [Training Checkpointing](https://awesome-repositories.com/f/artificial-intelligence-ml/training-checkpointing.md) — Persists intermediate training data, model checkpoints, and metric logs to local or distributed filesystems. ([source](https://github.com/horovod/horovod/blob/master/docs/spark.rst))

### Data & Databases

- [Collective Communication Operations](https://awesome-repositories.com/f/data-databases/collective-communication-operations.md) — Implements critical communication patterns like all-reduce and all-gather to synchronize gradients across distributed GPUs. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))
- [Backend Selection](https://awesome-repositories.com/f/data-databases/collective-communication-operations/backend-selection.md) — Selects specific backend implementations for communication operations based on available network hardware and vendor optimizations. ([source](https://github.com/horovod/horovod/blob/master/docs/gpus.rst))
- [Tensor Communication Batching](https://awesome-repositories.com/f/data-databases/batch-processing/batch-matrix-multiplication-utilities/query-batching/tensor-communication-batching.md) — Groups multiple small tensors into larger buffers to reduce network overhead during gradient synchronization.
- [Collective Operation Isolation](https://awesome-repositories.com/f/data-databases/collective-communication-operations/collective-operation-isolation.md) — Runs distinct collective communication tasks on specific subsets of worker processes to enable concurrent operations. ([source](https://github.com/horovod/horovod/blob/master/docs/process_set.rst))
- [Parallel Data Loaders](https://awesome-repositories.com/f/data-databases/data-pipeline-orchestration/data-engineering-pipelines/parallel-data-loaders.md) — Implements parallel shuffling and memory caching for large datasets to maximize data throughput. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))

### DevOps & Infrastructure

- [Distributed Script Launchers](https://awesome-repositories.com/f/devops-infrastructure/distributed-script-launchers.md) — Launches training scripts across multiple GPUs on a single machine or across a cluster of multiple machines. ([source](https://github.com/horovod/horovod/blob/master/docs/mpi.rst))
- [Distributed Task Orchestration](https://awesome-repositories.com/f/devops-infrastructure/distributed-task-orchestration.md) — Orchestrates the launch of distributed tasks by managing worker registration and execution across the cluster. ([source](https://github.com/horovod/horovod/blob/master/docs/contributors.rst))
- [MPI Cluster Orchestrators](https://awesome-repositories.com/f/devops-infrastructure/managed-cluster-orchestration/mpi-cluster-orchestrators.md) — Detects hostnames and GPU resources within a cluster to initialize MPI-based distributed training processes. ([source](https://github.com/horovod/horovod/blob/master/docs/lsf.rst))
- [ML Infrastructure Managers](https://awesome-repositories.com/f/devops-infrastructure/ml-infrastructure-managers.md) — Provides infrastructure management for deploying and scaling distributed deep learning environments on Kubernetes.
- [Multi-node Orchestration](https://awesome-repositories.com/f/devops-infrastructure/multi-node-orchestration.md) — Coordinates multiple containers across different machines to scale model training across distributed nodes. ([source](https://github.com/horovod/horovod/blob/master/docs/docker.rst))
- [Resource Allocation](https://awesome-repositories.com/f/devops-infrastructure/resource-allocation.md) — Allocates specific CPU and GPU hardware resources to training processes to optimize hardware utilization. ([source](https://github.com/horovod/horovod/blob/master/docs/spark.rst))
- [Training Orchestrators](https://awesome-repositories.com/f/devops-infrastructure/worker-node-management/distributed-orchestration/training-orchestrators.md) — Coordinates distributed training jobs with the ability to dynamically adjust worker counts and recover from node failures. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))
- [Elastic Training Scaling](https://awesome-repositories.com/f/devops-infrastructure/worker-scaling/elastic-training-scaling.md) — Allows adjusting the number of active workers at runtime and recovering from failed hosts without stopping training.
- [Custom Container Images](https://awesome-repositories.com/f/devops-infrastructure/container-orchestration/image-management-tools/custom-container-images.md) — Generates container images tailored to specific deep learning frameworks and GPU drivers using configurable build arguments. ([source](https://github.com/horovod/horovod/tree/master/docker))
- [Containerized Training Environments](https://awesome-repositories.com/f/devops-infrastructure/deployment-management-strategies/execution-platforms-and-targets/deployment-environments/containerized-training-environments.md) — Provides pre-configured container images to standardize the training environment across different hardware configurations. ([source](https://github.com/horovod/horovod/blob/master/docs/docker.rst))
- [Kubernetes Job Orchestration](https://awesome-repositories.com/f/devops-infrastructure/kubernetes-deployments/kubernetes-application-deployments/kubernetes-job-orchestration.md) — Bootstraps distributed deep learning environments using Kubernetes statefulsets and jobs for workers and drivers. ([source](https://github.com/horovod/horovod/tree/master/docker/helm/))
- [Ray Cluster Integrations](https://awesome-repositories.com/f/devops-infrastructure/ray-cluster-integrations.md) — Distributes deep learning workloads across Ray clusters by wrapping stateful processes as actors. ([source](https://github.com/horovod/horovod/blob/master/docs/ray.rst))
- [Worker Groupings](https://awesome-repositories.com/f/devops-infrastructure/worker-node-management/worker-groupings.md) — Allows running collective operations on specific groups of workers to isolate communication and reduce overhead. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))

### Networking & Communication

- [Distributed Coordination Primitives](https://awesome-repositories.com/f/networking-communication/distributed-systems-p2p/distributed-systems-coordination/distributed-coordination-primitives.md) — Coordinates execution between process ranks using barriers to ensure all workers reach a consistent state. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))
- [Communication Compression](https://awesome-repositories.com/f/networking-communication/data-compression/communication-compression.md) — Reduces volume of data transferred between nodes using half-precision compression for gradients and weights. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))
- [RDMA Networking](https://awesome-repositories.com/f/networking-communication/rdma-networking.md) — Integrates network interface cards via memory registration to enable low-latency direct memory access between nodes. ([source](https://github.com/horovod/horovod/blob/master/docs/docker.rst))

### Software Engineering & Architecture

- [Accelerator Device Mapping](https://awesome-repositories.com/f/software-engineering-architecture/state-mapping/infrastructure-resource-mapping/accelerator-device-mapping.md) — Maps training processes to specific CPU or GPU devices to optimize hardware utilization based on cluster allocation.
- [Master-Worker Coordination](https://awesome-repositories.com/f/software-engineering-architecture/distributed-coordination-systems/task-coordinations/master-worker-coordination.md) — Uses rank-based identifiers to coordinate barriers and broadcasting between a master process and worker nodes.

### Scientific & Mathematical Computing

- [MPI Communication](https://awesome-repositories.com/f/scientific-mathematical-computing/high-performance-execution-environments/high-performance-and-parallel-computing/high-performance-computing/mpi-communication.md) — Uses MPI collective communication protocols like all-reduce and all-gather to synchronize parallel training tasks.

### System Administration & Monitoring

- [GPU Profilers](https://awesome-repositories.com/f/system-administration-monitoring/gpu-profilers.md) — Analyzes GPU hardware utilization and kernel execution performance during distributed training. ([source](https://github.com/horovod/horovod/blob/master/CHANGELOG.md))
- [Execution Tracing](https://awesome-repositories.com/f/system-administration-monitoring/monitoring-and-observability/execution-tracing-analysis/execution-tracing.md) — Records the sequence of worker activities and tensor operations to analyze synchronization efficiency. ([source](https://github.com/horovod/horovod/blob/master/docs/timeline.rst))

### Testing & Quality Assurance

- [Distributed Training Timelines](https://awesome-repositories.com/f/testing-quality-assurance/performance-testing-analysis/performance-analysis/distributed-training-timelines.md) — Records detailed timelines of communication and computation events to identify bottlenecks in distributed training. ([source](https://github.com/horovod/horovod#readme))
- [Distributed Scalability Analysis](https://awesome-repositories.com/f/testing-quality-assurance/performance-testing-analysis/performance-diagnostics/performance-measurement/distributed-scalability-analysis.md) — Provides tools to evaluate scalability and throughput using synthetic or real datasets to baseline performance and diagnose distributed bottlenecks. ([source](https://github.com/horovod/horovod/blob/master/docs/benchmarks.rst))

### Part of an Awesome List

- [Deep Learning Frameworks](https://awesome-repositories.com/f/awesome-lists/ai/deep-learning-frameworks.md) — Supports distributed training across multiple deep learning frameworks.
- [MLOps and Infrastructure](https://awesome-repositories.com/f/awesome-lists/ai/mlops-and-infrastructure.md) — Distributed training framework for major frameworks.
- [Optimization Tools](https://awesome-repositories.com/f/awesome-lists/ai/optimization-tools.md) — Framework for distributed deep learning training across clusters.
- [Computation and Optimization](https://awesome-repositories.com/f/awesome-lists/devtools/computation-and-optimization.md) — Distributed training framework for TensorFlow, Keras, and PyTorch.
- [Parallel Programming Frameworks](https://awesome-repositories.com/f/awesome-lists/devtools/parallel-programming-frameworks.md) — Distributed deep learning training framework for major ML libraries.