Aibrix

Aibrix est un orchestrateur d'inférence conçu pour la mise à l'échelle, le routage et la gestion du déploiement de modèles de langage étendus à travers des clusters vLLM distribués. Il sert de passerelle centralisée pour l'équilibrage de charge et le routage du trafic vers des répliques et des versions de modèles spécifiques.

Le système gère l'efficacité des ressources via un autoscaler de cluster GPU qui ajuste le nombre d'instances de calcul en fonction du volume de requêtes en temps réel. Il optimise davantage les opérations en mélangeant différents types d'accélérateurs au sein d'un même cluster et en utilisant un orchestrateur d'adaptateurs de modèles pour déployer des adaptateurs de paramètres légers sur des modèles de base partagés.

Les capacités étendues incluent l'utilisation d'un gestionnaire de cache clé-valeur distribué pour partager les données de jetons à travers les moteurs d'inférence et l'implémentation de la surveillance de la santé du matériel pour détecter les défaillances des unités de traitement. Le projet fournit également un pipeline de métriques unifié pour standardiser la collecte de données de performance à travers divers environnements d'exécution.

Features

Inference Gateways - Provides a centralized gateway that routes incoming inference requests to specific model replicas and versions for load balancing.
Inference Deployment Orchestrators - Provides a system for managing the scaling and deployment of LLM inference workloads across distributed vLLM clusters.
Adapter-Based Model Deployment - Implements a system for deploying lightweight parameter adapters on shared base models to optimize hardware resource usage.
Traffic Management - Directs requests across multiple models and replicas through a centralized gateway to balance load.
Inference Traffic Routing - Directs inference requests across multiple model replicas and versions through a centralized gateway to balance load.
Automatic Inference Workload Scaling - Automatically adjusts the number of inference serving replicas based on real-time demand and request volume.
Large Language Model Serving - Spreads massive model weights and computation across multiple hardware nodes to handle high request volumes.
Auto Scaling - Adjusts compute instance counts in real time based on request volume to maintain target response times.
Adapter Management - Manages the dynamic loading and serving of lightweight adapters to run multiple model variants on shared hardware.
Distributed Model Parallelism - Spreads large model weights and computation across multiple hardware nodes to handle massive parameter counts.
Heterogeneous Hardware Scaling - Mixes different GPU and accelerator types within a single cluster to optimize operational costs and service levels.
KV Cache Management - Shares token data across different inference engines using a distributed key-value cache to increase capacity.
Inference Cost Optimizers - Optimizes operational costs by mixing different hardware types for inference while meeting service level objectives.
Multi-GPU Workload Distribution - Distributes computationally heavy LLM workloads across multiple GPU nodes to increase total throughput.
LLM KV Cache Stores - Utilizes a distributed key-value cache manager to share token data across inference engines and increase throughput.
GPU Resource Optimization - Optimizes operational costs by mixing different accelerator types and monitoring hardware health within a single cluster.
Cluster Autoscalers - Adjusts compute instance counts in real time based on request volume and response time targets.

Historique des stars

vllm-projectaibrix

Name: vllm-project/aibrix
Author: vllm-project

View on GitHub

4,882 stars607 forksGoApache-2.02 vues

Aibrix

Features

Inference Gateways - Provides a centralized gateway that routes incoming inference requests to specific model replicas and versions for load balancing.
Inference Deployment Orchestrators - Provides a system for managing the scaling and deployment of LLM inference workloads across distributed vLLM clusters.
Adapter-Based Model Deployment - Implements a system for deploying lightweight parameter adapters on shared base models to optimize hardware resource usage.
Traffic Management - Directs requests across multiple models and replicas through a centralized gateway to balance load.
Inference Traffic Routing - Directs inference requests across multiple model replicas and versions through a centralized gateway to balance load.
Automatic Inference Workload Scaling - Automatically adjusts the number of inference serving replicas based on real-time demand and request volume.
Large Language Model Serving - Spreads massive model weights and computation across multiple hardware nodes to handle high request volumes.
Auto Scaling - Adjusts compute instance counts in real time based on request volume to maintain target response times.
Adapter Management - Manages the dynamic loading and serving of lightweight adapters to run multiple model variants on shared hardware.
Distributed Model Parallelism - Spreads large model weights and computation across multiple hardware nodes to handle massive parameter counts.
Heterogeneous Hardware Scaling - Mixes different GPU and accelerator types within a single cluster to optimize operational costs and service levels.
KV Cache Management - Shares token data across different inference engines using a distributed key-value cache to increase capacity.
Inference Cost Optimizers - Optimizes operational costs by mixing different hardware types for inference while meeting service level objectives.
Multi-GPU Workload Distribution - Distributes computationally heavy LLM workloads across multiple GPU nodes to increase total throughput.
LLM KV Cache Stores - Utilizes a distributed key-value cache manager to share token data across inference engines and increase throughput.
GPU Resource Optimization - Optimizes operational costs by mixing different accelerator types and monitoring hardware health within a single cluster.
Cluster Autoscalers - Adjusts compute instance counts in real time based on request volume and response time targets.

Alternatives open source à Aibrix

Projets open source similaires, classés selon le nombre de fonctionnalités partagées avec Aibrix.

llm-d/llm-d
llm-d/llm-d
2,514Voir sur GitHub
llm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov
Shell
Voir sur GitHub2,514
kserve/kserve
kserve/kserve
5,576Voir sur GitHub
KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere
Go
Voir sur GitHub5,576
vllm-project/semantic-router
vllm-project/semantic-router
3,205Voir sur GitHub
Goai-gatewaybert-classificationfine-tuning
Voir sur GitHub3,205
kubeflow/kfserving
kubeflow/kfserving
5,576Voir sur GitHub
KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines inference services as custom resources with declarative YAML specifications, enabling a Kubernetes-native approach to model deployment and lifecycle management. The platform leverages Knative-based serverless scaling for automatic scale-to-zero and revision management, and supports a pluggable serving runtime architecture that maps model formats to containerized execution environments. KServe distinguishes itself through model-aware autoscaling that scales replicas based on token
Go
Voir sur GitHub5,576

Voir les 30 alternatives à Aibrix

Questions fréquentes

Que fait vllm-project/aibrix ?

Quelles sont les fonctionnalités principales de vllm-project/aibrix ?

Les fonctionnalités principales de vllm-project/aibrix sont : Inference Gateways, Inference Deployment Orchestrators, Adapter-Based Model Deployment, Traffic Management, Inference Traffic Routing, Automatic Inference Workload Scaling, Large Language Model Serving, Auto Scaling.

Quelles sont les alternatives open-source à vllm-project/aibrix ?

Les alternatives open-source à vllm-project/aibrix incluent : llm-d/llm-d — llm-d is a distributed serving framework designed for large language model inference. It functions as an inference… kserve/kserve — KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference… vllm-project/semantic-router. kubeflow/kfserving — KServe is an open platform for deploying and serving generative and predictive AI models on Kubernetes. It defines… thu-pacman/chitu — Chitu is a distributed serving platform and orchestrator for large language model inference. It functions as a compute… pytorch/serve — This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production…