awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
vllm-project avatar

vllm-project/aibrix

0
View on GitHub↗

Aibrix

Aibrix is an inference orchestrator designed for scaling, routing, and managing the deployment of large language models across distributed vLLM clusters. It serves as a centralized gateway for load balancing and routing traffic to specific model replicas and versions.

The system manages resource efficiency through a GPU cluster autoscaler that adjusts compute instance counts based on real-time request volume. It further optimizes operations by mixing different accelerator types within a single cluster and utilizing a model adapter orchestrator to deploy lightweight parameter adapters on shared base models.

Broad capabilities include the use of a distributed key-value cache manager to share token data across inference engines and the implementation of hardware health monitoring to detect processing unit failures. The project also provides a unified metrics pipeline to standardize performance data collection across diverse runtime environments.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Inference Gateways - Provides a centralized gateway that routes incoming inference requests to specific model replicas and versions for load balancing.
  • Inference Deployment Orchestrators - Provides a system for managing the scaling and deployment of LLM inference workloads across distributed vLLM clusters.
  • Adapter-Based Model Deployment - Implements a system for deploying lightweight parameter adapters on shared base models to optimize hardware resource usage.
  • Traffic Management - Directs requests across multiple models and replicas through a centralized gateway to balance load.
  • Inference Traffic Routing - Directs inference requests across multiple model replicas and versions through a centralized gateway to balance load.
  • Automatic Inference Workload Scaling - Automatically adjusts the number of inference serving replicas based on real-time demand and request volume.
  • Large Language Model Serving - Spreads massive model weights and computation across multiple hardware nodes to handle high request volumes.
  • Auto Scaling - Adjusts compute instance counts in real time based on request volume to maintain target response times.
  • Adapter Management - Manages the dynamic loading and serving of lightweight adapters to run multiple model variants on shared hardware.
  • Distributed Model Parallelism - Spreads large model weights and computation across multiple hardware nodes to handle massive parameter counts.
  • Heterogeneous Hardware Scaling - Mixes different GPU and accelerator types within a single cluster to optimize operational costs and service levels.
  • KV Cache Management - Shares token data across different inference engines using a distributed key-value cache to increase capacity.
  • Inference Cost Optimizers - Optimizes operational costs by mixing different hardware types for inference while meeting service level objectives.
  • Multi-GPU Workload Distribution - Distributes computationally heavy LLM workloads across multiple GPU nodes to increase total throughput.
  • LLM KV Cache Stores - Utilizes a distributed key-value cache manager to share token data across inference engines and increase throughput.
  • GPU Resource Optimization - Optimizes operational costs by mixing different accelerator types and monitoring hardware health within a single cluster.
  • Cluster Autoscalers - Adjusts compute instance counts in real time based on request volume and response time targets.
  • High-Performance Serving Systems - Cost-efficient Kubernetes-native control plane for distributed GenAI inference.
4,882 stars·607 forks·Go·Apache-2.0·39 views

Star history

Star history chart for vllm-project/aibrixStar history chart for vllm-project/aibrix

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Projects sharing features with Aibrix

These projects share indexed features with Aibrix. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • llm-d/llm-dllm-d avatar

    llm-d/llm-d

    5,167View on GitHub↗

    llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov

    Python
    View on GitHub↗5,167
  • kserve/kservekserve avatar

    kserve/kserve

    5,576View on GitHub↗

    KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

    Go
    View on GitHub↗5,576
  • sgl-project/sglangsgl-project avatar

    sgl-project/sglang

    38,561View on GitHub↗

    Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It provides a programmable interface for orchestrating complex generation workflows, enabling developers to coordinate multi-turn dialogues, tool invocations, and reasoning chains through a domain-specific language. The platform is built to support production-scale deployments, offering an OpenAI-compatible API that allows for integration with existing application ecosystems. The system distinguishes itself through a disaggregated architecture that separates compute-intensive pr

    Pythonattentionblackwellcuda
    View on GitHub↗38,561
  • lmcache/lmcacheLMCache avatar

    LMCache/LMCache

    12,307View on GitHub↗

    LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model inference. It functions as a tiered storage layer that offloads tensors from GPU memory to CPU RAM, local disks, or remote object stores, enabling the reuse of cached prefixes across different inference sessions and serving engines. The system differentiates itself through a disaggregated prefill-decode model, which separates prompt processing from token generation by transferring caches between distributed compute nodes. It utilizes peer-to-peer orchestration to share and retrieve

    Pythonamdcudafast
    View on GitHub↗12,307
Compare all 30 related projects→

Frequently asked questions

What does vllm-project/aibrix do?

Aibrix is an inference orchestrator designed for scaling, routing, and managing the deployment of large language models across distributed vLLM clusters. It serves as a centralized gateway for load balancing and routing traffic to specific model replicas and versions.

What are the main features of vllm-project/aibrix?

The main features of vllm-project/aibrix are: Inference Gateways, Inference Deployment Orchestrators, Adapter-Based Model Deployment, Traffic Management, Inference Traffic Routing, Automatic Inference Workload Scaling, Large Language Model Serving, Auto Scaling.

Which projects share features with vllm-project/aibrix?

Projects with overlapping indexed features include: llm-d/llm-d — llm-d is a distributed serving framework designed for large language model inference. It functions as an inference… kserve/kserve — KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference… sgl-project/sglang — Sglang is a high-performance inference engine and serving system designed for large language and multimodal models. It… lmcache/lmcache — LMCache is a distributed key-value cache manager and tiering system designed to accelerate large language model… vllm-project/semantic-router. kubeflow/kfserving — KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines…