Projects sharing features with GuidedQuant

The official implementation of the EMNLP 2023 paper LLM-FP4

MLSys 2024 Best Paper Award AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

ICML 2023 SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Microsoft Automatic Mixed Precision Library

An acceleration library that supports arbitrary bit-width combinatorial quantization operations

This project is a quantized fine-tuning framework for large language models. It implements a low-rank adaptation library and a four-bit quantizer to reduce the GPU memory requirements needed to train large models. The framework utilizes four-bit quantization and low-rank adapters to enable model training on consumer-grade hardware. It further reduces the memory footprint through double quantization and a paged optimizer that offloads states to system RAM. The system supports distributed training across multiple GPUs to handle larger parameter scales and includes utilities for custom dataset

This method identifies a binary weight or binary weight and activation subnetwork within a randomly initialized network that achieves performance comparable to, and sometimes better than, a weight-optimized network. The resulting binarized and pruned networks that achieve comparable performance…