Awesome GitHub RepositoriesModel Quantization Frameworks

vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

Burn is a deep learning framework designed for building, training, and deploying neural networks using a modular architecture. As a machine learning library built in Rust, it provides a backend-agnostic computational engine that enables the execution of models across diverse hardware, including central processors, graphics processors, and web runtimes. The framework distinguishes itself through a highly portable design that allows developers to maintain a single workflow for both training and inference across heterogeneous environments. It incorporates advanced optimization techniques such as

ggml is a low-level C++ tensor library and machine learning inference engine designed for performing mathematical operations on multi-dimensional arrays across diverse hardware platforms. It provides a foundational toolset for executing machine learning models and calculating mathematical gradients through an automatic differentiation library. The project features a quantized tensor framework that converts floating-point weights into integer representations to reduce memory usage and increase inference speed. It utilizes a custom binary format for model serialization to ensure rapid loading a

This project is a comprehensive toolkit designed for the full lifecycle management of large language and multimodal models. It functions as a unified orchestrator that handles the entire development process, ranging from dataset preparation and supervised fine-tuning to advanced reinforcement learning alignment and production-ready inference deployment. The platform distinguishes itself through a specialized reinforcement learning library that supports complex optimization algorithms, including group relative policy optimization and leave-one-out techniques, to improve model instruction-follo

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This project is a comprehensive technical reference and educational resource focused on the lifecycle of large language models. It provides structured learning materials that cover the foundational mechanics of transformer architectures, the mathematical principles of attention mechanisms, and the engineering practices required for modern generative artificial intelligence. The repository serves as a guide for both technical skill development and professional preparation, offering a curriculum that spans from model training and inference optimization to advanced alignment techniques. It detai

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

gpt-fast 是一个 PyTorch Transformer 推理引擎，专为使用原生张量库实现的文本生成而设计。它提供了一个运行时环境，用于执行大语言模型，而无需外部 C++ 扩展。 该项目实现了推测解码，通过使用小型草稿模型进行 Token 预测和较大模型进行验证来加速生成。它通过编译后的预填充阶段和跨多个图形处理单元分片线性层的多 GPU 张量并行库进一步优化了性能。 内存效率通过支持 int8 和 int4 权重以及分组张量量化的量化运行时进行管理。该系统还包括用于架构参数化、文本分词和使用标准化工具进行模型准确性评估的工具。

llama-rs 是一个用 Rust 实现的本地大语言模型推理引擎。它支持在本地硬件上执行模型计算，以根据用户提示生成文本响应。 该项目利用基于 Rust 的张量运算和直接内存模型映射来处理高性能线性代数和高效的权重加载。它结合了权重量化，通过将高精度权重转换为较小格式来减小模型的内存占用。 该系统包括用于交互式聊天会话和一次性提示的命令行界面，以及用于保存和恢复对话历史的文件备份会话持久化。它还提供用于从远程中心检索分词器配置的实用程序，以及用于计算困惑度分数以评估模型性能的工具。

h2o-llmstudio 是一个语言模型训练框架，提供了一个用于在自定义数据集上微调大型语言模型的无代码图形界面。它作为一个专门的工具，用于管理训练生命周期，从配置超参数到监控性能指标。 该项目通过一个多 GPU 训练编排器脱颖而出，该编排器通过数据并行处理分发工作负载，并提供了一个用于内存高效微调的低秩适应（LoRA）工具。它还包括一个模型评估仪表板，具有交互式聊天界面，以验证对话性能和响应质量。 该平台涵盖了广泛的功能面，包括带有模式映射的数据集准备、用于减少内存占用的模型量化，以及用于比较训练运行的实验管理。它还提供了用于本地模型导出和发布到社区模型中心的实用程序。 该系统包括一个用于触发实验和在自动化工作流中管理输出文件的命令行界面。

fastllm 是一套用于模型权重转换、混合专家 (MoE) 运行时和张量并行的专用软件组件。它提供了一个兼容 OpenAI 的 API 服务器，通过标准化的请求格式公开大语言模型功能。 该项目具有一个张量并行框架，可将计算工作负载拆分到多个 GPU 上以加速执行。它包含一个针对混合专家架构优化的专用运行时，以及一个将模型权重转换为低精度格式以减少内存使用并提高吞吐量的量化工具。 系统涵盖了分布式推理的高级工作流，包括设备映射内存管理、动态批处理和混合模式执行。它还提供了一个用于模型管理和部署配置的命令行界面和终端用户界面。

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

Doctor-Dignity is a privacy-preserving medical AI framework designed to execute large language models and diagnostic reasoning tasks locally on edge hardware. It provides a local inference engine and retrieval augmented generation implementation that ensures sensitive health data remains offline by removing dependencies on external cloud servers and internet connectivity. The project includes a medical fine-tuning framework for adapting base language models to specialized clinical domains using parameter-efficient methods. To enable execution on resource-constrained and mobile devices, it pro

llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of large language models. It provides a framework for compressing models using weight and activation quantization to enable more efficient deployment. The project distinguishes itself through a distributed quantization framework that utilizes data-parallel processing and disk-based weight offloading to handle massive model checkpoints that exceed available system memory. It includes specialized compressors for diverse architectures, including Mixture-of-Experts, Vision-Language,

This framework provides a toolkit for fine-tuning large language models by combining distributed data parallelism with parameter sharding and quantization techniques. It is designed to scale the training of massive neural networks across multiple graphics processors, enabling the execution of models that exceed the memory capacity of individual hardware units. The library distinguishes itself by integrating low-rank adaptation with memory-efficient weight loading and quantization-aware parameter sharding. By initializing model weights directly on the graphics processor and applying granular l