1 repository
Awesome GitHub RepositoriesInference Batch Packing
Techniques for packing multiple model inference requests into a single GPU forward pass to maximize throughput.
Distinct from Request Batching: Distinct from Request Batching: specifically addresses grouping different model adapters into a single GPU execution cycle for LLM inference.
Explore 1 awesome GitHub repository matching data & databases · Inference Batch Packing. Refine with filters or upvote what's useful.
Awesome Inference Batch Packing GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- predibase/lorax
predibase/lorax
3,724Vezi pe GitHubLorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Maximizes aggregate throughput by packing requests for different adapters into a single GPU forward pass.
Pythonfine-tuninggptllama
