1 repository
Awesome GitHub RepositoriesDisaggregated Request Routing
Routing logic specifically designed to dispatch requests between separate prefill and decode engine instances.
Distinct from Prefill-Decode Disaggregation: Focuses on the routing mechanism between disaggregated phases, whereas the parent focuses on the architectural separation itself.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Disaggregated Request Routing. Refine with filters or upvote what's useful.
Awesome Disaggregated Request Routing GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- paddlepaddle/fastdeploy
PaddlePaddle/FastDeploy
3,700Vezi pe GitHubFastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc
Routes incoming requests to optimal pairs of prefill and decode instances to maximize hardware utilization and throughput.
Pythonernieernie-45ernie-45-vl
