1 مستودع
Awesome GitHub RepositoriesAdapter Offloading
Asynchronous prefetching and moving of adapter weights between CPU and GPU memory.
Distinct from Memory Offloading Frameworks: Distinct from general tensor offloading: specifically manages the lifecycle of multiple LoRA adapters.
Explore 1 awesome GitHub repository matching operating systems & systems programming · Adapter Offloading. Refine with filters or upvote what's useful.
Awesome Adapter Offloading GitHub Repositories
اعثر على أفضل المستودعات باستخدام الذكاء الاصطناعي.سنبحث عن أفضل المستودعات المطابقة باستخدام الذكاء الاصطناعي.
- predibase/lorax
predibase/lorax
3,724عرض على GitHubLorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It functions as a high-throughput system capable of deploying models via Kubernetes and managing the dynamic swapping of Low-Rank Adaptation adapters per request. The server distinguishes itself through multi-adapter dynamic batching, which allows requests using different adapter weights to be processed in a single GPU forward pass. It employs just-in-time adapter loading and weighted adapter merging to maximize throughput and enable multi-tasking without sacrificing performance.
Optimizes throughput by asynchronously prefetching and offloading adapters between GPU and CPU memory.
Pythonfine-tuninggptllama
