AutoGPTQ

AutoGPTQ es un kit de herramientas de compresión de modelos y un framework de cuantización post-entrenamiento diseñado para reducir la huella de memoria de modelos de lenguaje grandes. Utiliza el algoritmo GPTQ para comprimir los pesos de las redes neuronales, reduciendo los requisitos de hardware y el uso de VRAM.

El proyecto sirve como un acelerador de inferencia al proporcionar kernels optimizados que aumentan la velocidad de generación de tokens. Cuenta con extensibilidad de arquitectura de modelo, permitiendo que las capacidades de cuantización se añadan a nuevas estructuras de modelos mediante patrones configurables.

El framework cubre una tubería de cuantización integral, incluyendo compresión de pesos por capa, estimación de escala basada en calibración y mapeo de memoria específico de precisión. También incluye sistemas para la evaluación del rendimiento del modelo para medir el impacto de la cuantización en la precisión en tareas de lenguaje y resumen.

Features

GPTQ Quantization Libraries - Implements the GPTQ algorithm for high-efficiency post-training weight quantization of large language models.
LLM Inference GPU Kernel Libraries - Provides a library of high-performance GPU kernels for accelerating token generation in large language models.
Inference Optimization Kernels - Uses specialized computational kernels to accelerate the token generation and decoding phases of quantized LLMs.
Calibration-Driven Quantization - Implements calibration-driven quantization to determine optimal scaling factors using representative data samples.
Neural Network Model Compression - Reduces the memory footprint and computational complexity of trained neural networks through quantization.
Model Compression - Reduces the size and computational requirements of large language models through weight quantization.
Quantized Inference Accelerators - Executes neural network computations in reduced precision to speed up token generation on GPUs.
Weight Quantization - Compresses model weights into lower-precision formats to reduce memory footprint and accelerate inference.
Post-Training Quantization - Provides a pipeline for reducing model precision after training is complete without requiring additional training cycles.
Inference Kernels - Provides hand-optimized CUDA kernels specifically designed to accelerate the inference phase of quantized neural networks.
Quantization Architecture Support - Allows the quantization pipeline to be extended to support new and custom neural network architectures.
Sequential Layer Compression - Processes quantization sequentially across model layers to preserve numerical stability and output accuracy.
Model Optimization - Quantization package for LLMs based on GPTQ.

Historial de estrellas

AutoGPTQAutoGPTQArchived

Name: autogptq/autogptq
Author: AutoGPTQ

View on GitHub

5,070 estrellas541 forksPythonMIT5 vistas

AutoGPTQ

Features

GPTQ Quantization Libraries - Implements the GPTQ algorithm for high-efficiency post-training weight quantization of large language models.
LLM Inference GPU Kernel Libraries - Provides a library of high-performance GPU kernels for accelerating token generation in large language models.
Inference Optimization Kernels - Uses specialized computational kernels to accelerate the token generation and decoding phases of quantized LLMs.
Calibration-Driven Quantization - Implements calibration-driven quantization to determine optimal scaling factors using representative data samples.
Neural Network Model Compression - Reduces the memory footprint and computational complexity of trained neural networks through quantization.
Model Compression - Reduces the size and computational requirements of large language models through weight quantization.
Quantized Inference Accelerators - Executes neural network computations in reduced precision to speed up token generation on GPUs.
Weight Quantization - Compresses model weights into lower-precision formats to reduce memory footprint and accelerate inference.
Post-Training Quantization - Provides a pipeline for reducing model precision after training is complete without requiring additional training cycles.
Inference Kernels - Provides hand-optimized CUDA kernels specifically designed to accelerate the inference phase of quantized neural networks.
Quantization Architecture Support - Allows the quantization pipeline to be extended to support new and custom neural network architectures.
Sequential Layer Compression - Processes quantization sequentially across model layers to preserve numerical stability and output accuracy.
Model Optimization - Quantization package for LLMs based on GPTQ.

Alternativas open-source a AutoGPTQ

Proyectos open-source similares, clasificados según cuántas características comparten con AutoGPTQ.

vllm-project/llm-compressor
vllm-project/llm-compressor
2,764Ver en GitHub
llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of large language models. It provides a framework for compressing models using weight and activation quantization to enable more efficient deployment. The project distinguishes itself through a distributed quantization framework that utilizes data-parallel processing and disk-based weight offloading to handle massive model checkpoints that exceed available system memory. It includes specialized compressors for diverse architectures, including Mixture-of-Experts, Vision-Language,
Pythoncompressionquantizationsparsity
Ver en GitHub2,764
intel/neural-compressor
intel/neural-compressor
2,585Ver en GitHub
Neural Compressor is a deep learning model compression toolkit and AI inference acceleration engine. It functions as an automated model quantization tool and hardware-aware model compiler designed to reduce the memory footprint of neural networks and decrease execution latency. The project provides specialized frameworks for optimizing large language models, utilizing weight-only quantization and hardware-specific kernels to improve the operational efficiency of generative AI workloads. It maps neural network operators to specialized CPU and GPU vector instructions to accelerate model executi
Pythonauto-tuningawqfp4
Ver en GitHub2,585
panqiwei/autogptq
PanQiWei/AutoGPTQ
5,073Ver en GitHub
AutoGPTQ is a model compression framework designed to reduce the memory footprint and increase the inference speed of large language models. It utilizes the GPTQ algorithm to compress model weights, allowing these models to run on hardware with limited VRAM. The toolkit provides an architecture quantization pipeline that supports the integration of custom model classes for various neural network architectures. It includes a mixed-precision inference engine with optimized kernels to accelerate matrix multiplication during deployment. The framework covers the full weight-compression workflow,
Python
Ver en GitHub5,073
internlm/lmdeploy
InternLM/lmdeploy
7,903Ver en GitHub
lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models. It functions as a multi-modal model server and compression toolkit designed to serve models with high throughput and low latency. The system enables the distribution of model services across multiple machines using request-based load balancing and tensor parallelism. It includes specialized tools for model quantization and compression to reduce the memory footprint of weights and caches. The framework covers broad capability areas including production deployment, distributed
Pythoncodellamacuda-kernelsdeepspeed
Ver en GitHub7,903

Ver las 30 alternativas a AutoGPTQ

Preguntas frecuentes

¿Qué hace autogptq/autogptq?

¿Cuáles son las características principales de autogptq/autogptq?

Las características principales de autogptq/autogptq son: GPTQ Quantization Libraries, LLM Inference GPU Kernel Libraries, Inference Optimization Kernels, Calibration-Driven Quantization, Neural Network Model Compression, Model Compression, Quantized Inference Accelerators, Weight Quantization.

¿Qué alternativas de código abierto existen para autogptq/autogptq?

Las alternativas de código abierto para autogptq/autogptq incluyen: vllm-project/llm-compressor — llm-compressor is a quantization toolkit and post-training library designed to reduce the memory footprint and size of… intel/neural-compressor — Neural Compressor is a deep learning model compression toolkit and AI inference acceleration engine. It functions as… panqiwei/autogptq — AutoGPTQ is a model compression framework designed to reduce the memory footprint and increase the inference speed of… internlm/lmdeploy — lmdeploy is a high-performance inference engine and deployment framework for large language models and vision models.… nunchaku-ai/comfyui-nunchaku — ComfyUI-nunchaku is a 4-bit diffusion inference engine and a set of nodes for running low-precision quantized… tingsongyu/pytorch-tutorial-2nd — This project is a comprehensive instructional resource and course for building neural networks using PyTorch. It…

AutoGPTQ

Features

Historial de estrellas

AutoGPTQ

Features

Alternativas open-source a AutoGPTQ

vllm-project/llm-compressor

intel/neural-compressor

PanQiWei/AutoGPTQ

InternLM/lmdeploy

Preguntas frecuentes

Historial de estrellas

Alternativas open-source a AutoGPTQ

vllm-project/llm-compressor

intel/neural-compressor

PanQiWei/AutoGPTQ

InternLM/lmdeploy

Preguntas frecuentes