1 Repo
Awesome GitHub RepositoriesRuntime Quantizers
Tools that apply quantization to a model already loaded in memory, reducing memory and compute during execution.
Distinct from Model Quantization: Distinct from Model Quantization: focuses on applying quantization to a running model rather than offline or load-time conversion.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Runtime Quantizers. Refine with filters or upvote what's useful.
Awesome Runtime Quantizers GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- ericlbuehler/mistral.rs
EricLBuehler/mistral.rs
6,597Auf GitHub ansehenmistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe
Applies a chosen quantization format to a loaded model during execution, reducing memory and compute requirements.
Rustllmrustuqff
