mostlygeek/llama-swap
Llama Swap
Llama-swap is a local inference orchestrator and API gateway for large language models. It functions as an OpenAI API proxy that manages the lifecycle of multiple local model servers, automatically starting and stopping them to swap models based on incoming request identifiers.
The project distinguishes itself through dynamic model swapping and hardware optimization. It utilizes a specialized matrix-based concurrency control to define which models can run simultaneously and employs cost-based eviction to remove inactive servers from memory based on relative resource costs.
The system provides comprehensive model management, including identifier aliasing, request filtering, and the execution of lifecycle commands for containers or virtual machines. It also includes observability tools such as a visual model testing playground, real-time system performance monitoring, and security features like API key verification and TLS encryption.
Configuration updates are handled via dynamic reloading that monitors the filesystem for changes without requiring a manual restart.
Features
- Dynamic Model Swapping - Automatically switches active AI model servers on demand by extracting model identifiers from incoming requests.
- Model API Gateways - Provides a translation layer that exposes local model capabilities through standardized API endpoints.
- Local Inference Orchestrators - Coordinates the lifecycle and configuration of multiple local model servers through a unified system.
- Local Model Management - Provides tools for hosting and configuring multiple large language models on local hardware.
- Local Model Orchestrators - Orchestrates the lifecycle, resource allocation, and concurrent execution of multiple local large language models.
- Demand-Based Server Swapping - Loads and unloads compatible model servers based on request demand to manage hardware resources efficiently.
- OpenAI-Compatible Model Servers - Provides an HTTP endpoint compatible with the OpenAI API specification for seamless integration of local models.
- Hardware Concurrency Matrices - Uses a specialized configuration matrix to define which models can run simultaneously on available hardware.
- Model Swappers - Automatically manages the lifecycle of llama.cpp servers to swap models based on incoming request identifiers.
- Model Server Evictions - Optimizes memory by prioritizing the shutdown of inactive models based on assigned resource costs.
- OpenAI-Compatible API Servers - Implements an OpenAI-compatible REST API interface to route requests to various local inference backends.
- Server Instance Swapping - Manages model transitions by starting and stopping entire server instances instead of reloading weights in-process.
- VRAM Efficiency Strategies - Maximizes GPU and CPU memory efficiency through automated model eviction and idle timeouts.
- Runtime Model Swapping - Switches between different generative AI model servers on demand to optimize hardware utilization.
- Model Unloading Policies - Removes inactive models from memory after a specific timeout period to free up system resources.
- Inference API Proxies - Functions as a proxy specifically designed to route and optimize requests directed at local model inference APIs.
- Automated Memory Eviction - Assigns relative resource costs to models to prioritize which ones to evict when memory is limited.
- Concurrent Model Execution - Executes multiple different models in memory simultaneously using unique port assignments.
- Inference Backend Lifecycles - Manages the startup and shutdown of containerized or virtualized model servers using shell commands.
- Model Request Proxies - Acts as a gateway that intercepts and routes API requests to interchangeable local language model providers.
- Concurrency Coordination Matrices - Uses a specialized swap matrix configuration to define which models can run simultaneously on available hardware.
- API Proxies - Implements a middleware layer that translates and forwards API requests to ensure compatibility with multiple AI API specifications.
- Model Server Controls - Controls the startup and shutdown of model servers using commands for containers and virtual machines.
- Model Management - Manages model swapping for local OpenAI-compatible inference servers.
Star history
How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
