Awesome GitHub RepositoriesQuantization Evaluation

MiniCPM is a collection of small language models designed for local, on-device deployment in resource-constrained environments. The project focuses on running dense Transformer models on consumer hardware, including GPUs, CPUs, and Apple Silicon, without requiring custom code forks. The project distinguishes itself through heavy optimization for edge hardware, utilizing quantized weight compression in GGUF and MLX formats to reduce memory overhead. It implements advanced inference techniques such as speculative sampling and radix-tree prefix caching to accelerate generation speed and throughp

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a config-driven system for instantiating components, orchestrating distributed training, and managing parameter-efficient fine-tuning with quantization support, all through YAML-based configurations and command-line overrides. The library distinguishes itself through its comprehensive post-training workflow orchestration, combining supervised fine-tuning, preference optimization (DPO, PPO, GRPO), knowledge distillation, and quantization-aware training in a single configurable pip

Torchtune is a PyTorch-native library for fine-tuning, aligning, and quantizing large language models. It provides a configurable training pipeline orchestrated through YAML recipes, with CLI overrides and component swapping, distributed training via FSDP2, memory optimizations, and parameter-efficient fine-tuning methods like LoRA, DoRA, and QLoRA. The library distinguishes itself through its YAML-driven configuration system that defines all training parameters and instantiates components from config files, with full CLI override capability for any field or component at launch time. It suppo

AutoGPTQ ist ein Framework zur Modellkomprimierung, das entwickelt wurde, um den Speicherbedarf zu reduzieren und die Inferenzgeschwindigkeit großer Sprachmodelle zu erhöhen. Es nutzt den GPTQ-Algorithmus zur Komprimierung von Modellgewichten, wodurch diese Modelle auf Hardware mit begrenztem VRAM ausgeführt werden können. Das Toolkit bietet eine Architektur-Quantisierungspipeline, die die Integration benutzerdefinierter Modellklassen für verschiedene neuronale Netzwerkarchitekturen unterstützt. Es enthält eine Mixed-Precision-Inferenz-Engine mit optimierten Kernels, um die Matrixmultiplikation während des Deployments zu beschleunigen. Das Framework deckt den gesamten Workflow der Gewichtskomprimierung ab, von der Kalibrierung und Quantisierung bis hin zur Genauigkeitsbewertung nachgelagerter Aufgaben. Diese Tools messen den Performance-Verlust durch den Vergleich der Ausgaben quantisierter Modelle mit den Originalgewichten bei Benchmark-Aufgaben.