1 个仓库
Awesome GitHub RepositoriesLoad-Aware Request Routing
Dynamic routing of requests based on real-time resource utilization and cache load of backend workers.
Distinct from Worker-Task Routing: Focuses on load-based balancing for inference servers rather than general label-based or affinity-based task routing.
Explore 1 awesome GitHub repository matching development tools & productivity · Load-Aware Request Routing. Refine with filters or upvote what's useful.
Awesome Load-Aware Request Routing GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- ai-dynamo/dynamo
ai-dynamo/dynamo
6,112在 GitHub 上查看Dynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Routes inference requests to the least-loaded worker based on active cache utilization.
Rust
