1 repo
Awesome GitHub RepositoriesHierarchical Request Routing
Architectural patterns for routing requests through multiple layers of control to specialized resource pools.
Distinct from Concurrent Request Pools: Unlike general object pooling or network request pools, this focuses on high-level routing logic for AI workload classes.
Explore 1 awesome GitHub repository matching software engineering & architecture · Hierarchical Request Routing. Refine with filters or upvote what's useful.
Awesome Hierarchical Request Routing GitHub Repositories
Find the best repos with AI.We'll search the best matching repositories with AI.
- ai-dynamo/dynamo
ai-dynamo/dynamo
6,112View on GitHubDynamo is a distributed inference orchestration platform designed for large language models. It functions as a system to coordinate prefill and decode phases across GPU nodes, utilizing a multi-backend runtime adapter to connect engines like vLLM and TensorRT-LLM through a unified block-oriented memory interface. An OpenAI-compatible API server provides the frontend for integration with existing tools and clients. The project is distinguished by its disaggregated serving architecture, which separates prompt processing and token generation onto independent GPU pools to optimize throughput and
Routes requests through a control layer to separate pools optimized for different request classes.
Rust
