1 个仓库
Awesome GitHub RepositoriesInference Batch Packing
Techniques for packing multiple model inference requests into a single GPU forward pass to maximize throughput.
Distinct from Request Batching: Distinct from Request Batching: specifically addresses grouping different model adapters into a single GPU execution cycle for LLM inference.
Explore 1 awesome GitHub repository matching data & databases · Inference Batch Packing. Refine with filters or upvote what's useful.
Awesome Inference Batch Packing GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- predibase/lorax
predibase/lorax
3,724在 GitHub 上查看Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Maximizes aggregate throughput by packing requests for different adapters into a single GPU forward pass.
Pythonfine-tuninggptllama
