1 repository
Awesome GitHub RepositoriesServer Instance Swapping
Replacing active model servers by stopping one process and starting another to transition weights.
Distinct from Server Process Management: Focuses on the act of swapping between distinct server processes rather than general process management
Explore 1 awesome GitHub repository matching web development · Server Instance Swapping. Refine with filters or upvote what's useful.
Awesome Server Instance Swapping GitHub Repositories
Găsește cele mai bune repo-uri cu AI.Vom căuta cele mai potrivite repository-uri folosind AI.
- mostlygeek/llama-swap
mostlygeek/llama-swap
4,786Vezi pe GitHubLlama-swap is a local inference orchestrator and API gateway for large language models. It functions as an OpenAI API proxy that manages the lifecycle of multiple local model servers, automatically starting and stopping them to swap models based on incoming request identifiers. The project distinguishes itself through dynamic model swapping and hardware optimization. It utilizes a specialized matrix-based concurrency control to define which models can run simultaneously and employs cost-based eviction to remove inactive servers from memory based on relative resource costs. The system provide
Manages model transitions by starting and stopping entire server instances instead of reloading weights in-process.
Go
