Awesome GitHub RepositoriesTensor Memory Mapping

This project provides Rust bindings for the TensorFlow C API, serving as a tensor computation interface and machine learning library. It enables the construction and execution of machine learning models and neural networks by bridging a systems language to high-performance backends. The framework supports GPU-accelerated computing to increase the speed of model training and inference by offloading mathematical operations to graphics processing units. It offers both graph-based computation for defining static network architectures and an eager execution mode for immediate operation calls durin

AutoGPTQ 是一个模型压缩工具包和训练后量化框架，旨在减少大语言模型的内存占用。它利用 GPTQ 算法压缩神经网络权重，降低硬件要求并减少 VRAM 使用量。 该项目通过提供优化内核来提高 Token 生成速度，从而充当推理加速器。它具有模型架构扩展性，允许通过可配置模式将量化能力添加到新的模型结构中。 该框架涵盖了全面的量化流水线，包括层级权重压缩、基于校准的缩放估计以及特定精度的内存映射。它还包括用于模型性能评估的系统，以衡量量化对语言和摘要任务准确性的影响。