Awesome GitHub RepositoriesLow-Bit Weight Quantization

FlexLLMGen is an inference engine and runtime designed to run large language models on a single GPU by combining weight compression with tensor offloading. It reduces model weight memory usage by approximately 70% through 4-bit quantization, and stores model parameters, attention cache, and hidden states across GPU, CPU, and disk to fit models larger than available GPU memory. The project distinguishes itself through a throughput-oriented batching approach that processes multiple generation requests together in large batches to maximize throughput on a single GPU. It also supports distributed

BigDL is a PyTorch acceleration framework and distributed inference engine designed for large language models. It provides a toolkit for running models on Intel hardware, integrating quantization tools and libraries for parameter-efficient fine-tuning. The project distinguishes itself through the use of pipeline parallelism to distribute model workloads across multiple hardware accelerators. It utilizes low-bit integer quantization and speculative decoding to reduce memory footprints and decrease text generation latency. The system covers broad capabilities in model optimization, including w

Intel XPU LLM Acceleration Library is a toolkit designed to accelerate large language model inference and finetuning on Intel CPUs, GPUs, and NPUs. It provides a distributed inference engine for scaling models across multiple accelerators, a multimodal model runtime for vision and speech tasks, and a low-bit model quantization tool for converting weights into INT4, FP8, and GGUF formats. The project features a parameter-efficient finetuning framework that enables model adaptation using QLoRA and DPO on Intel hardware. It distinguishes itself by providing specialized optimizations for Intel XP

Yi is a bilingual language model and foundation model designed for natural language processing, reasoning, and reading comprehension in both English and Chinese. It is built as a transformer-based architecture capable of general purpose text generation and conversational tasks. The model is distinguished by its ability to function as a long context system, processing and analyzing extended input sequences up to 200k tokens. It also supports quantized versions that use low-bit precision to reduce memory footprints, enabling execution on consumer-grade hardware. The project covers a broad rang

mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

Lit-llama هو إطار عمل تنفيذ يعتمد على PyTorch لنموذج اللغة LLaMA، ويوفر نظاماً للتدريب المسبق، والضبط الدقيق، والاستدلال عالي الأداء. يتضمن خط أنابيب تدريب مسبق لإنشاء نماذج لغوية أساسية من الصفر وأدوات لتشغيل الأوزان المدربة مسبقاً لتوليد نص طبيعي والتنبؤ بالتسلسلات. يوفر المشروع مجموعات أدوات متخصصة للضبط الدقيق الفعال للمعلمات باستخدام التكيف منخفض الرتبة (LoRA) والمحولات خفيفة الوزن. كما يتضمن مكتبة تكميم (quantization) تقلل من بصمات ذاكرة النموذج من خلال دقة 4 بت و8 بت لتمكين التنفيذ على الأجهزة ذات الموارد المحدودة. يدمج إطار العمل تصميم محول مبسط ويوظف انتباه الفلاش (flash attention) لتحسين الذاكرة والسرعة. كما يدير مجموعات بيانات واسعة النطاق من خلال تنسيقات بيانات البث لتجنب تحميل مجموعات النصوص الكاملة في ذاكرة النظام.

ACE Step 1.5 is a local text-to-music generation and audio editing system that runs on consumer hardware. It transforms plain-language descriptions into full-length songs with lyrics, and can edit existing audio through cover generation, vocal removal, track separation, and selective repainting. The system supports multilingual prompts and lyrics in over 50 languages, and provides precise control over musical structure including duration, BPM, key, and time signature. The project distinguishes itself through a dual-stream diffusion architecture that processes separate latent streams for vocal

Baichuan-7B is an open-source 7 billion parameter bilingual Transformer model designed for text generation and few-shot learning across Chinese and English. It is built on a large Transformer architecture trained on a bilingual corpus, enabling it to produce coherent text in both languages from a single model. The model incorporates several optimization techniques that distinguish it from standard large language models. It uses rotary position embeddings that can extrapolate to longer sequences than seen during training, allowing context extension beyond the original 4096-token training lengt

AutoGPTQ هو إطار عمل لضغط النماذج مصمم لتقليل بصمة الذاكرة وزيادة سرعة الاستنتاج لنماذج اللغات الكبيرة. يستخدم خوارزمية GPTQ لضغط أوزان النموذج، مما يسمح لهذه النماذج بالعمل على أجهزة ذات ذاكرة فيديو (VRAM) محدودة. توفر مجموعة الأدوات خط أنابيب لتكميم البنية (Architecture quantization) يدعم دمج فئات النماذج المخصصة لبنيات الشبكات العصبية المختلفة. يتضمن محرك استنتاج بدقة مختلطة مع نواة محسنة لتسريع ضرب المصفوفات أثناء النشر. يغطي إطار العمل سير عمل ضغط الأوزان بالكامل، من المعايرة والتكميم إلى تقييم الدقة اللاحق. تقيس هذه الأدوات فقدان الأداء من خلال مقارنة مخرجات النماذج المكممة مقابل الأوزان الأصلية في مهام القياس المعياري.

AutoGPTQ هو مجموعة أدوات لضغط النماذج وإطار عمل للتكميم بعد التدريب مصمم لتقليل بصمة الذاكرة لنماذج اللغات الكبيرة. يستخدم خوارزمية GPTQ لضغط أوزان الشبكة العصبية، مما يقلل من متطلبات الأجهزة ويقلل من استخدام ذاكرة الفيديو (VRAM). يعمل المشروع كمسرع للاستنتاج من خلال توفير نواة محسنة تزيد من سرعة توليد الرموز (Tokens). يتميز بقابلية توسيع بنية النموذج، مما يسمح بإضافة قدرات التكميم إلى هياكل النماذج الجديدة من خلال أنماط قابلة للتكوين. يغطي إطار العمل خط أنابيب تكميم شامل، بما في ذلك ضغط الأوزان على مستوى الطبقة، وتقدير النطاق القائم على المعايرة، وتعيين الذاكرة الخاص بالدقة. كما يتضمن أنظمة لتقييم أداء النموذج لقياس تأثير التكميم على الدقة عبر مهام اللغة والتلخيص.

SakuraLLM is a multi-format document translation system that hosts large language models for translating Japanese text into other languages. It functions as an inference server that exposes translation models through an OpenAI-compatible API, allowing any tool supporting the OpenAI client format to send translation requests. The system is designed as a glossary-aware translation engine that applies user-defined term dictionaries to ensure consistent translation of proper nouns and names across outputs. The project distinguishes itself by supporting multiple high-performance inference backends

Baichuan2 is a collection of pre-trained large language models, including base and chat variants, designed for natural language generation and multi-turn conversational AI. It provides an inference engine and a fine-tuning framework to adapt these models to custom datasets and specialized domains. The project features a quantization toolkit and an inference engine that enable model execution across diverse hardware, including graphics processors, central processors, and specialized accelerators. These tools support low-bit weight quantization to reduce memory usage and increase inference spee

Nunchaku is a 4-bit model quantization library and diffusion model inference engine designed to run large-scale neural networks on consumer GPUs. It functions as a GPU-accelerated optimizer that reduces VRAM usage and increases inference speed through weight compression and memory management. The project utilizes low-rank weight decomposition and SVD weight quantization to compress models to four-bit precision while maintaining visual fidelity. It employs kernel-level operator fusion to minimize data movement and hardware-aware precision mapping to adjust numerical precision based on the unde