1 dépôt
Awesome GitHub RepositoriesLoad-Aware Request Routing
Dynamic routing of requests based on real-time resource utilization and cache load of backend workers.
Distinct from Worker-Task Routing: Focuses on load-based balancing for inference servers rather than general label-based or affinity-based task routing.
Explore 1 awesome GitHub repository matching development tools & productivity · Load-Aware Request Routing. Refine with filters or upvote what's useful.
Awesome Load-Aware Request Routing GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- ai-dynamo/dynamo
ai-dynamo/dynamo
6,112Voir sur GitHubDynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Routes inference requests to the least-loaded worker based on active cache utilization.
Rust
