1 dépôt
Awesome GitHub RepositoriesAsynchronous Gating
Flow-control mechanisms that queue and dispatch offline inference requests based on hardware availability.
Distinct from Inference Batching: Distinct from Inference Batching: focuses on the queuing and gating of asynchronous jobs rather than the grouping of requests for a single compute pass.
Explore 1 awesome GitHub repository matching data & databases · Asynchronous Gating. Refine with filters or upvote what's useful.
Awesome Asynchronous Gating GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- llm-d/llm-d
llm-d/llm-d
2,514Voir sur GitHubllm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov
Implements flow-control gating to execute offline batch requests using spare hardware capacity.
Shell
