1 个仓库
Awesome GitHub RepositoriesRuntime Quantizers
Tools that apply quantization to a model already loaded in memory, reducing memory and compute during execution.
Distinct from Model Quantization: Distinct from Model Quantization: focuses on applying quantization to a running model rather than offline or load-time conversion.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Runtime Quantizers. Refine with filters or upvote what's useful.
Awesome Runtime Quantizers GitHub Repositories
用 AI 发现最棒的仓库。我们将通过 AI 为您搜索最匹配的仓库。
- ericlbuehler/mistral.rs
EricLBuehler/mistral.rs
6,597在 GitHub 上查看mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe
Applies a chosen quantization format to a loaded model during execution, reducing memory and compute requirements.
Rustllmrustuqff
