1 Repo
Awesome GitHub RepositoriesQuantization Recommenders
Tools that suggest optimal quantization methods and device mappings for a given model and hardware.
Distinct from Quantization Methods: Distinct from Quantization Methods: focuses on recommending settings, not the techniques themselves.
Explore 1 awesome GitHub repository matching artificial intelligence & ml · Quantization Recommenders. Refine with filters or upvote what's useful.
Awesome Quantization Recommenders GitHub Repositories
Finde die besten Repos mit KI.Wir suchen mit KI nach den am besten passenden Repositories.
- ericlbuehler/mistral.rs
EricLBuehler/mistral.rs
6,597Auf GitHub ansehenmistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe
Suggests an optimal quantization method and device mapping for a given model, optionally biased by a target quantization.
Rustllmrustuqff
