Awesome GitHub RepositoriesLow-Bit Weight Quantization

FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory. The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed

BigDL 是一个 PyTorch 加速框架和分布式推理引擎，专为大语言模型设计。它提供了一个在 Intel 硬件上运行模型的工具包，集成了量化工具和用于参数高效微调的库。 该项目通过使用流水线并行将模型工作负载分布在多个硬件加速器上而脱颖而出。它利用低位整数量化和推测解码来减少内存占用并降低文本生成延迟。 该系统涵盖了模型优化的广泛功能，包括权重压缩和量化模型加载。它还支持硬件加速的训练例程，以使预训练模型适应特定任务。

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Lit-llama is a PyTorch-based implementation framework for the LLaMA language model, providing a system for pre-training, fine-tuning, and high-performance inference. It includes a pre-training pipeline for creating foundational language models from scratch and tools for running pretrained weights to generate natural text and predict sequences. The project provides specialized toolkits for parameter-efficient fine-tuning using low-rank adaptation and lightweight adapters. It also includes a quantization library that reduces model memory footprints through four-bit and eight-bit precision to en

ACE Step 1.5 is a local text-to-music generation and audio editing system that runs on consumer hardware. It transforms plain-language descriptions into full-length songs with lyrics, and can edit existing audio through cover generation, vocal removal, track separation, and selective repainting. The system supports multilingual prompts and lyrics in over 50 languages, and provides precise control over musical structure including duration, BPM, key, and time signature. The project distinguishes itself through a dual-stream diffusion architecture that processes separate latent streams for vocal

Baichuan-7B is an open-source 7 billion parameter bilingual Transformer model designed for text generation and few-shot learning across Chinese and English. It is built on a large Transformer architecture trained on a bilingual corpus, enabling it to produce coherent text in both languages from a single model. The model incorporates several optimization techniques that distinguish it from standard large language models. It uses rotary position embeddings that can extrapolate to longer sequences than seen during training, allowing context extension beyond the original 4096-token training lengt

AutoGPTQ 是一个模型压缩框架，旨在减少大语言模型的内存占用并提高推理速度。它利用 GPTQ 算法压缩模型权重，使这些模型能够在 VRAM 有限的硬件上运行。 该工具包提供了一个架构量化流水线，支持为各种神经网络架构集成自定义模型类。它包含一个带有优化内核的混合精度推理引擎，可在部署期间加速矩阵乘法。 该框架涵盖了从校准和量化到下游准确性评估的完整权重压缩工作流。这些工具通过在基准任务上比较量化模型与原始权重的输出来衡量性能损失。

AutoGPTQ 是一个模型压缩工具包和训练后量化框架，旨在减少大语言模型的内存占用。它利用 GPTQ 算法压缩神经网络权重，降低硬件要求并减少 VRAM 使用量。 该项目通过提供优化内核来提高 Token 生成速度，从而充当推理加速器。它具有模型架构扩展性，允许通过可配置模式将量化能力添加到新的模型结构中。 该框架涵盖了全面的量化流水线，包括层级权重压缩、基于校准的缩放估计以及特定精度的内存映射。它还包括用于模型性能评估的系统，以衡量量化对语言和摘要任务准确性的影响。

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

Baichuan2 是一系列预训练大语言模型（包括基础版和对话版），专为自然语言生成和多轮对话 AI 设计。它提供了一个推理引擎和一个微调框架，用于将这些模型适配到自定义数据集和专业领域。 该项目具有量化工具包和推理引擎，支持在包括图形处理器、中央处理器和专用加速器在内的多种硬件上执行模型。这些工具支持低位权重量化，以减少内存使用并提高受限硬件上的推理速度。 该系统涵盖了广泛的能力，包括多机分布式训练、参数高效微调，以及用于人类交互的监督对齐。它还包括模型版本转换的实用工具，并通过命令行工具或基于 Web 的演示提供对话界面。

Nunchaku is a 4-bit model quantization library and diffusion model inference engine designed to run large-scale neural networks on consumer GPUs. It functions as a GPU-accelerated optimizer that reduces VRAM usage and increases inference speed through weight compression and memory management. The project utilizes low-rank weight decomposition and SVD weight quantization to compress models to four-bit precision while maintaining visual fidelity. It employs kernel-level operator fusion to minimize data movement and hardware-aware precision mapping to adjust numerical precision based on the unde