2 Repos
Awesome GitHub RepositoriesCommunication Layers
Software components that manage data exchange and synchronization between nodes in distributed machine learning systems.
Distinguishing note: Focuses on network-level communication and gradient synchronization rather than the training orchestration itself.
Explore 2 awesome GitHub repositories matching artificial intelligence & ml · Communication Layers. Refine with filters or upvote what's useful.
Awesome Communication Layers GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- deepspeedai/deepspeed
deepspeedai/DeepSpeed
42,528Auf GitHub ansehenDeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization
A communication layer that reduces network overhead during multi-node training through gradient compression and efficient parameter synchronization.
Pythonbillion-parameterscompressiondata-parallelism - horovod/horovod
horovod/horovod
14,686Auf GitHub ansehenHorovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi
Provides a communication abstraction layer that decouples the distributed training logic from specific network hardware and libraries.
Python
