NVIDIA/FasterTransformer

Features

• Inference and Serving - NVIDIA framework for accelerated LLM inference.
• Mixture of Experts - Optimizes MoE model execution for cloud-scale production.
• Model Quantization Tools - Optimized transformer implementation for cloud-scale production.
• Transformer Implementations - Optimized transformer implementation for high-performance inference.