1 Repo
Awesome GitHub RepositoriesOffline Batch Inference
Mechanisms for executing large-scale asynchronous model inference to maximize total hardware utilization.
Distinct from Batch Processing Utilities: Candidates focus on general data batching or training trials; this is specifically for asynchronous model inference.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Offline Batch Inference. Refine with filters or upvote what's useful.
Awesome Offline Batch Inference GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- llm-d/llm-d
llm-d/llm-d
2,514Auf GitHub ansehenllm-d is a distributed serving framework designed for large language model inference. It functions as an inference orchestrator and gateway, providing a control plane for deploying model replicas and managing hardware accelerators. The system includes a batch inference scheduler and a cache manager to coordinate request flow and memory utilization. The project is distinguished by a disaggregated serving architecture that separates prefill and decode execution phases across specialized workers to maximize throughput. It employs a hardware-agnostic control plane and tiered cache offloading, mov
Provides a mechanism to run large-scale asynchronous inference via compatible APIs to maximize total hardware utilization.
Shell
