1 Repo
Awesome GitHub RepositoriesInference Batch Packing
Techniques for packing multiple model inference requests into a single GPU forward pass to maximize throughput.
Distinct from Request Batching: Distinct from Request Batching: specifically addresses grouping different model adapters into a single GPU execution cycle for LLM inference.
Explore 1 awesome GitHub repository matching data & databases · Inference Batch Packing. Refine with filters or upvote what's useful.
Awesome Inference Batch Packing GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- predibase/lorax
predibase/lorax
3,724Auf GitHub ansehenLorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Maximizes aggregate throughput by packing requests for different adapters into a single GPU forward pass.
Pythonfine-tuninggptllama
