4 Repos
Awesome GitHub RepositoriesGradient Compression Techniques
Methods for quantizing and compressing gradient data to reduce network overhead during distributed training.
Distinguishing note: Specifically targets gradient transmission efficiency in distributed training, distinct from general data compression.
Explore 4 awesome GitHub repositories matching artificial intelligence & ml · Gradient Compression Techniques. Refine with filters or upvote what's useful.
Awesome Gradient Compression Techniques GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- microsoft/deepspeed
microsoft/DeepSpeed
42,533Auf GitHub ansehenDeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special
Quantizes gradients and weights during synchronization to reduce network traffic between distributed nodes.
Python - deepspeedai/deepspeed
deepspeedai/DeepSpeed
42,528Auf GitHub ansehenDeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization
Gradient data is compressed and quantized before network transmission to minimize bandwidth bottlenecks during large-scale distributed training sessions.
Pythonbillion-parameterscompressiondata-parallelism - apache/mxnet
apache/mxnet
20,829Auf GitHub ansehenThis project is a deep learning framework designed for constructing, training, and deploying neural networks across diverse hardware environments. It functions as a high-performance tensor computation library that provides both imperative and symbolic programming interfaces, allowing developers to balance flexible, step-by-step model building with the efficiency of compiled computation graphs. The framework distinguishes itself through a hybrid execution engine that integrates declarative graph compilation with imperative runtime logic. It supports scalable, distributed training across multip
Reduces network bandwidth requirements during distributed training by compressing gradient updates.
C++mxnet - horovod/horovod
horovod/horovod
14,686Auf GitHub ansehenHorovod is a distributed deep learning framework and gradient synchronizer designed to scale model training across multiple GPUs and compute nodes. It functions as a distributed training orchestrator and an elastic training engine, utilizing an MPI collective communication library to synchronize weights and gradients across TensorFlow, PyTorch, Keras, and MXNet models. The system distinguishes itself through dynamic elastic scaling, which allows it to adjust the number of active workers at runtime and recover from node failures. It optimizes communication efficiency using tensor fusion batchi
Reduces network bandwidth by compressing tensor gradients using half-precision formats during synchronization.
Python
