How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
ICML 2023 SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
MLSys 2024 Best Paper Award AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of massive AI models. It serves as a model parallelism orchestrator and a toolkit for scaling large language models across multiple GPUs and compute nodes. The project distinguishes itself through 3D parallelism orchestration, which combines data, pipeline, and tensor parallelism. It utilizes ZeRO-based memory partitioning to eliminate redundant storage and employs CPU-offload memory management to move weights and optimizer states to system RAM. Additionally, it provides special
This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset
Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".
The main features of ist-daslab/gptq are: Model Quantization, Model Quantization Tools, Quantization Frameworks.
Projects with overlapping indexed features include: mit-han-lab/smoothquant — [ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. microsoft/deepspeed — DeepSpeed is a distributed deep learning optimization library and framework designed for the training and inference of… mit-han-lab/llm-awq — [MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. nbasyl/llm-fp4 — The official implementation of the EMNLP 2023 paper LLM-FP4. artidoro/qlora — This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation… snu-mllab/guidedquant — Official PyTorch implementation of "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance"…