turboderp/exllamav2
Exllamav2
exllamav2 ist eine Hochleistungs-Inferenzbibliothek, die für das lokale Ausführen von Large Language Models auf Consumer-GPUs entwickelt wurde. Sie bietet einen GPU-beschleunigten Runner und Quantisierungstools, um die Modellausführung ohne Abhängigkeit von Cloud-Computing-Diensten zu ermöglichen.
Das Projekt verfügt über ein Quantisierungs-Dienstprogramm, das Modelle in gemischte Bitraten zwischen zwei und acht Bit komprimiert, um den VRAM-Bedarf zu reduzieren. Es zeichnet sich durch einen gebatchten Textgenerator aus, der gruppierte Anfragen verarbeitet und Cache-Daten dedupliziert, um den Durchsatz zu erhöhen.
Die Bibliothek deckt ein breites Funktionsspektrum ab, einschließlich asynchronem Token-Streaming für Echtzeit-Ausgabe, benutzerdefinierter GPU-Kernel-Ausführung für lineare Algebra-Operationen und lokalem Memory-Mapping für den Zugriff auf Modellgewichte mit geringer Latenz.
Features
- Local Inference Engines - Acts as a runtime optimized for executing large language models on consumer-grade hardware.
- Batch Inference Engines - Provides a system that processes multiple input requests simultaneously to increase text generation throughput.
- Batched Response Generation - Implements techniques for producing multiple model completions from a single prompt to increase throughput via batching.
- GPU-Accelerated Inference - Implements hardware acceleration techniques specifically applied to the inference phase of models on GPUs.
- Inference Libraries - Provides a high-performance library to load and execute large language models on local hardware.
- High-Throughput Text Inference - Delivers optimized model execution for high-volume text generation with low latency.
- Mixed-Precision Quantization - Compresses model weights into varying bit-widths between two and eight bits to reduce memory footprint.
- Model Quantization - Uses techniques to reduce the precision of model weights to decrease memory footprint and accelerate inference.
- LLM Quantization Frameworks - Ships a toolkit designed to implement and manage the quantization of large language model weights.
- Inference Batching - Groups multiple model inference requests into a single hardware execution pass to maximize GPU throughput.
- CUDA Compute Kernels - Employs custom CUDA kernels written in C++ to parallelize heavy mathematical computations for LLM inference.
- KV Cache Management - Optimizes inference efficiency by storing and reusing key-value pairs across multiple sequences in a batch.
- Memory-Mapped Weight Loaders - Maps model weight files directly into process memory to reduce RAM usage and improve load times.
- Incremental Inference Streaming - Provides a mechanism to send generated tokens to clients incrementally for real-time output.
- Token Stream Generators - Uses a generator interface to output each generated token immediately for real-time incremental display.
- Token Streamers - Provides components that deliver generated tokens incrementally to the user interface using asynchronous operations.
- Real-Time Text Streaming - Implements low-latency processing of continuous token streams for real-time AI application output.
- Consumer GPU Optimizations - Provides serving techniques tailored for running large models on consumer-grade hardware via precision reduction.
- Inference Engines - High-speed inference library for modern consumer-grade GPUs.
- Model Quantization - Fast inference library optimized for low-bitwidth model quantization.
