1 dépôt
Awesome GitHub RepositoriesMulti-Adapter Batching
Grouping requests that use different Low-Rank Adaptation weights into a single GPU forward pass.
Distinct from Inference Batching: Specifically handles batches with mixed adapter weights, whereas general inference batching assumes the same model weights.
Explore 1 awesome GitHub repository matching data & databases · Multi-Adapter Batching. Refine with filters or upvote what's useful.
Awesome Multi-Adapter Batching GitHub Repositories
Trouvez les meilleurs dépôts grâce à l'IA.Nous recherchons les dépôts les plus pertinents grâce à l'IA.
- predibase/lorax
predibase/lorax
3,724Voir sur GitHubLorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Processes requests using different LoRA adapters in a single GPU forward pass to maximize throughput.
Pythonfine-tuninggptllama
