Awesome GitHub RepositoriesModel Quantization Frameworks

vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

Burn is a deep learning framework designed for building, training, and deploying neural networks using a modular architecture. As a machine learning library built in Rust, it provides a backend-agnostic computational engine that enables the execution of models across diverse hardware, including central processors, graphics processors, and web runtimes. The framework distinguishes itself through a highly portable design that allows developers to maintain a single workflow for both training and inference across heterogeneous environments. It incorporates advanced optimization techniques such as

ggml is a low-level C++ tensor library and machine learning inference engine designed for performing mathematical operations on multi-dimensional arrays across diverse hardware platforms. It provides a foundational toolset for executing machine learning models and calculating mathematical gradients through an automatic differentiation library. The project features a quantized tensor framework that converts floating-point weights into integer representations to reduce memory usage and increase inference speed. It utilizes a custom binary format for model serialization to ensure rapid loading a

This project is a comprehensive toolkit designed for the full lifecycle management of large language and multimodal models. It functions as a unified orchestrator that handles the entire development process, ranging from dataset preparation and supervised fine-tuning to advanced reinforcement learning alignment and production-ready inference deployment. The platform distinguishes itself through a specialized reinforcement learning library that supports complex optimization algorithms, including group relative policy optimization and leave-one-out techniques, to improve model instruction-follo

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This project is a comprehensive technical reference and educational resource focused on the lifecycle of large language models. It provides structured learning materials that cover the foundational mechanics of transformer architectures, the mathematical principles of attention mechanisms, and the engineering practices required for modern generative artificial intelligence. The repository serves as a guide for both technical skill development and professional preparation, offering a curriculum that spans from model training and inference optimization to advanced alignment techniques. It detai

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library implementation. It provides a runtime for executing large language models without the need for external C++ extensions. The project implements speculative decoding to accelerate generation by using a small draft model for token prediction and a larger model for verification. It further optimizes performance through a compiled prefill stage and a multi-GPU tensor parallelism library that shards linear layers across multiple graphics processing units. Memory efficiency is managed throu

llama-rs ist eine Inferenz-Engine für lokale Large Language Models, die in Rust implementiert ist. Sie ermöglicht die Ausführung von Modellberechnungen auf lokaler Hardware, um Textantworten aus Benutzer-Prompts zu generieren. Das Projekt nutzt Rust-basierte Tensor-Operationen und Direct-Memory-Modell-Mapping, um High-Performance-Lineare-Algebra und effizientes Laden von Gewichten zu handhaben. Es integriert Weight-Quantization, um den Speicherbedarf von Modellen durch Konvertierung hochpräziser Gewichte in kleinere Formate zu reduzieren. Das System enthält ein Kommandozeilen-Interface für interaktive Chat-Sitzungen und einmalige Prompts, zusammen mit Datei-basierter Sitzungspersistenz, um Konversationsverläufe zu speichern und wiederherzustellen. Es bietet zudem Utilities zum Abrufen von Tokenizer-Konfigurationen von Remote-Hubs sowie Tools zur Berechnung von Perplexity-Scores, um die Modellleistung zu evaluieren.

h2o-llmstudio ist ein Framework für das Training von Sprachmodellen, das eine No-Code-Grafikschnittstelle für das Fine-Tuning großer Sprachmodelle auf benutzerdefinierten Datensätzen bietet. Es fungiert als spezialisiertes Tool für die Verwaltung des Trainings-Lebenszyklus, von der Konfiguration der Hyperparameter bis zur Überwachung von Leistungsmetriken. Das Projekt zeichnet sich durch einen Multi-GPU-Trainings-Orchestrator aus, der Workloads über Datenparallelverarbeitung verteilt, sowie durch ein Low-Rank-Adaptation-Tool für speichereffizientes Fine-Tuning. Es enthält zudem ein Modell-Evaluierungs-Dashboard mit einer interaktiven Chat-Schnittstelle, um die Konversationsleistung und Antwortqualität zu verifizieren. Die Plattform deckt eine breite Funktionsfläche ab, einschließlich Datensatzvorbereitung mit Schema-Mapping, Modell-Quantisierung zur Reduzierung des Speicherbedarfs und Experiment-Management für den Vergleich von Trainingsläufen. Sie bietet zudem Dienstprogramme für den lokalen Modell-Export und die Veröffentlichung in Community-Modell-Hubs. Das System enthält eine Befehlszeilenschnittstelle zum Auslösen von Experimenten und zur Verwaltung von Ausgabedateien innerhalb automatisierter Workflows.

fastllm is a set of specialized software components for model weight conversion, Mixture-of-Experts runtimes, and tensor parallelism. It provides an OpenAI compatible API server to expose large language model capabilities through a standardized request format. The project features a tensor parallelism framework that splits computational workloads across multiple GPUs to accelerate execution. It includes a dedicated runtime optimized for Mixture-of-Experts architectures and a quantization tool to convert model weights into lower precision formats to reduce memory usage and increase throughput.

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

Doctor-Dignity is a privacy-preserving medical AI framework designed to execute large language models and diagnostic reasoning tasks locally on edge hardware. It provides a local inference engine and retrieval augmented generation implementation that ensures sensitive health data remains offline by removing dependencies on external cloud servers and internet connectivity. The project includes a medical fine-tuning framework for adapting base language models to specialized clinical domains using parameter-efficient methods. To enable execution on resource-constrained and mobile devices, it pro

llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of large language models. It provides a framework for compressing models using weight and activation quantization to enable more efficient deployment. The project distinguishes itself through a distributed quantization framework that utilizes data-parallel processing and disk-based weight offloading to handle massive model checkpoints that exceed available system memory. It includes specialized compressors for diverse architectures, including Mixture-of-Experts, Vision-Language,

Dieses Framework bietet ein Toolkit für das Fine-Tuning großer Sprachmodelle durch die Kombination von verteilter Datenparallelität mit Parameter-Sharding und Quantisierungstechniken. Es wurde entwickelt, um das Training massiver neuronaler Netze über mehrere Grafikprozessoren hinweg zu skalieren und so die Ausführung von Modellen zu ermöglichen, die die Speicherkapazität einzelner Hardwareeinheiten übersteigen. Die Bibliothek zeichnet sich durch die Integration von Low-Rank-Adaption mit speichereffizientem Laden von Gewichten und quantisierungsbewusstem Parameter-Sharding aus. Durch die Initialisierung von Modellgewichten direkt auf dem Grafikprozessor und die Anwendung einer granularen, schichtweisen Umhüllung minimiert das Framework Speicherspitzen und reduziert den Kommunikationsaufwand während der verteilten Setup- und Trainingsphasen. Das System unterstützt das Training benutzerdefinierter Transformer-Architekturen durch flexible Wrapping-Richtlinien für Attention- und Multilayer-Perceptron-Schichten. Es optimiert die Ressourcennutzung weiter, indem es die numerische Präzision während der Berechnung dynamisch anpasst und so Trainingsstabilität gegen verfügbaren Hardwarespeicher abwägt. Das Projekt wird als Sammlung von Dienstprogrammen und Skripten vertrieben, die für den Einsatz in verteilten Rechenumgebungen vorgesehen sind.