Awesome GitHub RepositoriesModel Quantization Frameworks

vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware. The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach

This project is a distributed training infrastructure designed for aligning large language models through reinforcement learning. It functions as an end-to-end engine for complex alignment tasks, including proximal policy optimization, direct preference optimization, and iterative self-play. By providing a unified framework for multi-turn interactions and tool-use scenarios, it enables the development of models capable of reasoning and external environment engagement. The framework distinguishes itself through a decoupled architecture that separates model training from sample generation. This

Burn is a deep learning framework designed for building, training, and deploying neural networks using a modular architecture. As a machine learning library built in Rust, it provides a backend-agnostic computational engine that enables the execution of models across diverse hardware, including central processors, graphics processors, and web runtimes. The framework distinguishes itself through a highly portable design that allows developers to maintain a single workflow for both training and inference across heterogeneous environments. It incorporates advanced optimization techniques such as

ggml is a low-level C++ tensor library and machine learning inference engine designed for performing mathematical operations on multi-dimensional arrays across diverse hardware platforms. It provides a foundational toolset for executing machine learning models and calculating mathematical gradients through an automatic differentiation library. The project features a quantized tensor framework that converts floating-point weights into integer representations to reduce memory usage and increase inference speed. It utilizes a custom binary format for model serialization to ensure rapid loading a

This project is a comprehensive toolkit designed for the full lifecycle management of large language and multimodal models. It functions as a unified orchestrator that handles the entire development process, ranging from dataset preparation and supervised fine-tuning to advanced reinforcement learning alignment and production-ready inference deployment. The platform distinguishes itself through a specialized reinforcement learning library that supports complex optimization algorithms, including group relative policy optimization and leave-one-out techniques, to improve model instruction-follo

PaddleDetection is an object detection framework designed for the end-to-end development, training, and deployment of computer vision models. It provides a comprehensive library of modular neural network architectures and pipelines that support object detection, instance segmentation, and multi-object tracking tasks. The project distinguishes itself through a configuration-driven approach that decouples model components like backbones and heads, allowing for the flexible assembly of custom vision workflows. It incorporates advanced techniques such as anchor-free detection logic, joint detecti

This project is a comprehensive technical reference and educational resource focused on the lifecycle of large language models. It provides structured learning materials that cover the foundational mechanics of transformer architectures, the mathematical principles of attention mechanisms, and the engineering practices required for modern generative artificial intelligence. The repository serves as a guide for both technical skill development and professional preparation, offering a curriculum that spans from model training and inference optimization to advanced alignment techniques. It detai

Axolotl is a configuration-driven framework designed for the fine-tuning, evaluation, and quantization of large language models. It functions as a comprehensive orchestrator for distributed training, enabling users to manage complex workflows across multi-node and multi-GPU environments. By utilizing structured configuration files, the platform streamlines the setup of training parameters, dataset paths, and hardware distribution strategies. The project distinguishes itself through its support for diverse training methodologies, including full-parameter tuning, parameter-efficient adaptation,

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

gpt-fast is a PyTorch transformer inference engine designed for text generation using a native tensor library implementation. It provides a runtime for executing large language models without the need for external C++ extensions. The project implements speculative decoding to accelerate generation by using a small draft model for token prediction and a larger model for verification. It further optimizes performance through a compiled prefill stage and a multi-GPU tensor parallelism library that shards linear layers across multiple graphics processing units. Memory efficiency is managed throu

llama-rs هو محرك استنتاج للنماذج اللغوية الكبيرة محلياً تم تنفيذه بلغة Rust. يتيح تنفيذ حسابات النموذج على الأجهزة المحلية لتوليد استجابات نصية من مطالبات المستخدم. يستخدم المشروع عمليات الموتر القائمة على Rust وتعيين النموذج في الذاكرة المباشرة للتعامل مع الجبر الخطي عالي الأداء وتحميل الأوزان بكفاءة. يدمج تكميم الأوزان لتقليل أثر الذاكرة للنماذج عن طريق تحويل الأوزان عالية الدقة إلى تنسيقات أصغر. يتضمن النظام واجهة سطر أوامر لجلسات الدردشة التفاعلية والمطالبات الفردية، إلى جانب استمرارية الجلسة المدعومة بالملفات لحفظ واستعادة سجلات المحادثة. كما يوفر أدوات لاسترجاع تكوينات المقسم (tokenizer) من المراكز البعيدة وأدوات لحساب درجات الحيرة لتقييم أداء النموذج.

h2o-llmstudio هو إطار عمل لتدريب نماذج اللغة يوفر واجهة رسومية بدون كود لضبط النماذج اللغوية الكبيرة على مجموعات بيانات مخصصة. يعمل كأداة متخصصة لإدارة دورة حياة التدريب، من تكوين المعلمات الفائقة إلى مراقبة مقاييس الأداء. يتميز المشروع من خلال منسق تدريب متعدد وحدات معالجة الرسومات (multi-GPU) يوزع أعباء العمل عبر معالجة البيانات المتوازية وأداة تكيف منخفضة الرتبة (LoRA) للضبط الدقيق الموفر للذاكرة. كما يتضمن لوحة معلومات لتقييم النموذج تتميز بواجهة دردشة تفاعلية للتحقق من أداء المحادثة وجودة الاستجابة. تغطي المنصة سطح إمكانيات واسعاً بما في ذلك إعداد مجموعة البيانات مع تعيين المخطط، وتكميم النموذج لتقليل بصمات الذاكرة، وإدارة التجارب لمقارنة عمليات التدريب. كما توفر أدوات لتصدير النموذج المحلي والنشر في مراكز النماذج المجتمعية. يتضمن النظام واجهة سطر أوامر لتشغيل التجارب وإدارة ملفات المخرجات ضمن سير العمل الآلي.

fastllm is a set of specialized software components for model weight conversion, Mixture-of-Experts runtimes, and tensor parallelism. It provides an OpenAI compatible API server to expose large language model capabilities through a standardized request format. The project features a tensor parallelism framework that splits computational workloads across multiple GPUs to accelerate execution. It includes a dedicated runtime optimized for Mixture-of-Experts architectures and a quantization tool to convert model weights into lower precision formats to reduce memory usage and increase throughput.

CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models. The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

Doctor-Dignity is a privacy-preserving medical AI framework designed to execute large language models and diagnostic reasoning tasks locally on edge hardware. It provides a local inference engine and retrieval augmented generation implementation that ensures sensitive health data remains offline by removing dependencies on external cloud servers and internet connectivity. The project includes a medical fine-tuning framework for adapting base language models to specialized clinical domains using parameter-efficient methods. To enable execution on resource-constrained and mobile devices, it pro

llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of large language models. It provides a framework for compressing models using weight and activation quantization to enable more efficient deployment. The project distinguishes itself through a distributed quantization framework that utilizes data-parallel processing and disk-based weight offloading to handle massive model checkpoints that exceed available system memory. It includes specialized compressors for diverse architectures, including Mixture-of-Experts, Vision-Language,

This framework provides a toolkit for fine-tuning large language models by combining distributed data parallelism with parameter sharding and quantization techniques. It is designed to scale the training of massive neural networks across multiple graphics processors, enabling the execution of models that exceed the memory capacity of individual hardware units. The library distinguishes itself by integrating low-rank adaptation with memory-efficient weight loading and quantization-aware parameter sharding. By initializing model weights directly on the graphics processor and applying granular l