How this analysis was created: This summary and feature list were written by an AI model that read the project's README and public documentation pages. Each feature links to the documentation it came from; stars, license and language come straight from the GitHub API. The model does not read the source code, and the analysis is refreshed when the project is re-analysed. Learn more on our About page.
MLSys 2024 Best Paper Award AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special
Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".
This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset
[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
The main features of mit-han-lab/smoothquant are: Model Quantization, Model Quantization Tools, Quantization Frameworks.
Open-source alternatives to mit-han-lab/smoothquant include: mit-han-lab/llm-awq — [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. ist-daslab/gptq — Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers". microsoft/deepspeed — DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of… nbasyl/llm-fp4 — The official implementation of the EMNLP 2023 paper LLM-FP4. artidoro/qlora — This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation… snu-mllab/guidedquant — Official PyTorch implementation of "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance"…