How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov
KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere
Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr
LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve
Aibrix is an inference orchestrator designed for scaling, routing, and managing the deployment of large language models across distributed vLLM clusters. It serves as a centralized gateway for load balancing and routing traffic to specific model replicas and versions.
The main features of vllm-project/aibrix are: Inference Gateways, Inference Deployment Orchestrators, Adapter-Based Model Deployment, Traffic Management, Inference Traffic Routing, Automatic Inference Workload Scaling, Large Language Model Serving, Auto Scaling.
Projects with overlapping indexed features include: llm-d/llm-d — llm-d is a distributed serving framework designed for large language model inference. It functions as an inference… kserve/kserve — KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference… sgl-project/sglang — Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It… lmcache/lmcache — LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model… vllm-project/semantic-router. kubeflow/kfserving — KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines…