microsoft/DeepSpeed-MII

Features

• Inference and Serving - Low-latency inference engine powered by DeepSpeed.
• Model Serving Engines - Low-latency inference library powered by DeepSpeed optimizations.
• Inference Frameworks - Inference framework supporting load balancing and model quantization.