1 repo
Awesome GitHub RepositoriesHigh-Throughput Model Serving
Architectures designed to handle large volumes of concurrent inference requests with low latency.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · High-Throughput Model Serving. Refine with filters or upvote what's useful.
Awesome High-Throughput Model Serving GitHub Repositories
We'll search the best matching repositories with AI.
- vllm-project/vllm
vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token gen
Scales large language model inference to handle high volumes of concurrent requests with minimal latency.
Pythonamdblackwellcuda
