1 repo
Awesome GitHub RepositoriesMixture-of-Experts Inference Optimizers
Parallelization techniques and execution strategies for reducing latency in mixture-of-experts model inference.
Distinguishing note: Focuses on inference-time throughput and latency reduction for sparse models, distinct from training-time parallelism.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Mixture-of-Experts Inference Optimizers. Refine with filters or upvote what's useful.
Awesome Mixture-of-Experts Inference Optimizers GitHub Repositories
Find the best repos with AI.We'll search the best matching repositories with AI.
- deepspeedai/DeepSpeed
deepspeedai/DeepSpeed
41,638View on GitHubDeepSpeed is a high-performance library designed to scale deep learning model training and inference across massive clusters of GPUs and compute nodes. It provides a comprehensive suite of tools for distributed training, enabling the execution of models that exceed the memory capacity of single devices through advanced parameter partitioning, pipeline-based model parallelism, and memory-efficient state offloading. The framework distinguishes itself through specialized communication-efficient optimizers and hardware-aware acceleration techniques. By utilizing gradient compression, quantization
The framework achieves low latency and high throughput for mixture-of-experts models by using specialized parallelization techniques that avoid traditional dense model trade-offs.
Pythonbillion-parameterscompressiondata-parallelism
