1 dépôt
Awesome GitHub RepositoriesDisaggregated Request Routing
Routing logic specifically designed to dispatch requests between separate prefill and decode engine instances.
Distinct from Prefill-Decode Disaggregation: Focuses on the routing mechanism between disaggregated phases, whereas the parent focuses on the architectural separation itself.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Disaggregated Request Routing. Refine with filters or upvote what's useful.
Awesome Disaggregated Request Routing GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- paddlepaddle/fastdeploy
PaddlePaddle/FastDeploy
3,700Voir sur GitHubFastDeploy is a high-performance deployment framework for large language models, vision models, and multimodal models. It provides the infrastructure to launch model services that process combined image, video, and text inputs, exposing these capabilities through a standardized, OpenAI-compatible API for chat and text completions. The project distinguishes itself through advanced inference pipeline engineering and GPU optimization. It employs speculative decoding, tensor parallelism, and a disaggregated execution model that separates prefill and decode phases across different hardware resourc
Routes incoming requests to optimal pairs of prefill and decode instances to maximize hardware utilization and throughput.
Pythonernieernie-45ernie-45-vl
