awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
turboderp-org avatar

turboderp-org/exllamav2

0
View on GitHub↗

Exllamav2

exllamav2 is a high-performance inference engine and framework for executing large language models locally on consumer-class GPUs. It provides a complete system for local model deployment, including a specialized inference engine and tools for model quantization.

The project features a multi-GPU inference framework that distributes workloads across multiple graphics cards to run models that exceed the memory capacity of a single device. It includes a GPU model quantizer capable of converting models into mixed-precision formats between 2 and 8 bits to balance memory usage and accuracy.

The engine supports high-throughput text generation through batch-based parallel inference and asynchronous output streaming. These capabilities are supported by custom CUDA kernels and cache deduplication to optimize hardware utilization and reduce latency during token generation.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI

Features

  • Local Model Execution - Enables high-performance large language model inference on consumer-grade local hardware.
  • Local Inference Engines - Provides a high-performance local inference engine optimized for consumer-grade GPUs.
  • Multi-GPU Inference Runtimes - Provides a framework for distributing large language model workloads across multiple GPUs to bypass memory limits.
  • Mixed-Precision Quantization - Balances model size and accuracy using mixed-precision quantization between two and eight bits.
  • Mixed-Precision Quantizers - Provides a GPU model quantizer for converting models into flexible mixed-precision formats between 2 and 8 bits.
  • Multi-GPU Distribution - Splits large model layers across multiple graphics cards to execute models exceeding single-device memory.
  • Flexible Bitrate Quantization - Includes a utility to convert models into mixed quantization levels between 2 and 8 bits.
  • Batch Generation - Provides dynamic batch text generation to maximize throughput via smart caching and deduplication.
  • Text Generation Engines - Implements a high-performance text generation engine with smart caching for simultaneous prompt completions.
  • Inference Kernels - Uses hand-optimized custom CUDA kernels to accelerate the mathematical operations required for fast local inference.
  • Inference Batching - Executes multiple text completion prompts simultaneously using batch-based parallel inference to maximize GPU utilization.
  • Multi-GPU Sharding - Manages GPU memory by splitting model workloads across multiple graphics cards.
  • KV Cache Management - Increases throughput by sharing common prompt prefixes in memory via KV cache deduplication.
  • High-Throughput Text Inference - Optimizes text generation for high volume and throughput using smart caching and deduplication.
  • Real-Time Text Streaming - Sends generated model output incrementally via asynchronous processing for real-time user interaction.
  • Asynchronous Token Streaming - Processes model tokens in the background and streams them incrementally to reduce user-perceived latency.
  • Incremental Response Streaming - Implements incremental response streaming to provide immediate user feedback before the full sequence completes.
  • Model Quantization - Listed in the “Model Quantization” section of the Llm Course awesome list.
4,552 stars·337 forks·Python·MIT·21 views

Star history

Star history chart for turboderp-org/exllamav2Star history chart for turboderp-org/exllamav2

How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

Frequently asked questions

What does turboderp-org/exllamav2 do?

exllamav2 is a high-performance inference engine and framework for executing large language models locally on consumer-class GPUs. It provides a complete system for local model deployment, including a specialized inference engine and tools for model quantization.

What are the main features of turboderp-org/exllamav2?

The main features of turboderp-org/exllamav2 are: Local Model Execution, Local Inference Engines, Multi-GPU Inference Runtimes, Mixed-Precision Quantization, Mixed-Precision Quantizers, Multi-GPU Distribution, Flexible Bitrate Quantization, Batch Generation.

Which projects share features with turboderp-org/exllamav2?

Projects with overlapping indexed features include: turboderp/exllamav2 — exllamav2 is a high-performance inference library designed for running large language models locally on consumer-grade… modeltc/lightllm — LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a… mistralai/mistral-src — This project is a large language model inference library and framework designed to run models for text generation,… djdefrag/qualityscaler — QualityScaler is an AI video upscaler and local media processing tool designed to increase the resolution and visual… lightning-ai/litserve — LitServe is a Python AI inference server framework and LLM serving framework designed for high-concurrency inference.… llm-d/llm-d — llm-d is a distributed serving framework designed for large language model inference. It functions as an inference…

Projects sharing features with Exllamav2

These projects share indexed features with Exllamav2. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
  • turboderp/exllamav2turboderp avatar

    turboderp/exllamav2

    4,553View on GitHub↗

    exllamav2 is a high-performance inference library designed for running large language models locally on consumer-grade GPUs. It provides a GPU-accelerated runner and quantization tools to enable model execution without reliance on cloud-based computing services. The project features a quantization utility that compresses models into mixed bitrates between two and eight bits to reduce video RAM requirements. It distinguishes itself through a batched text generator that handles grouped requests and deduplicates cache data to increase throughput. The library covers a broad capability surface in

    Python
    View on GitHub↗4,553
  • modeltc/lightllmModelTC avatar

    ModelTC/LightLLM

    3,901View on GitHub↗

    LightLLM is a high-performance serving framework for deploying and executing large language models. It functions as a multi-GPU inference engine and server capable of handling dense architectures, mixture-of-experts designs, and multimodal models that process both text and images. The system is distinguished by its specialized support for Mixture-of-Experts models using expert parallelism and fused kernels. It implements structured text generation through deterministic state machines and pushdown automata to enforce precise output formats. To optimize throughput, the framework employs specula

    Pythondeep-learninggptllama
    View on GitHub↗3,901
  • mistralai/mistral-srcmistralai avatar

    mistralai/mistral-src

    10,821View on GitHub↗

    This project is a large language model inference library and framework designed to run models for text generation, problem solving, and coding assistance. It includes a multimodal framework for processing combined image and text inputs and a tool-use implementation that enables the execution of external functions based on model reasoning. The system features a distributed GPU inference engine that spreads large model workloads across multiple graphics processors to increase processing speed and meet memory requirements. It also provides containerized model deployment through pre-packaged imag

    Jupyter Notebook
    View on GitHub↗10,821
  • djdefrag/qualityscalerDjdefrag avatar

    Djdefrag/QualityScaler

    2,970View on GitHub↗

    QualityScaler is an AI video upscaler and local media processing tool designed to increase the resolution and visual quality of videos and images. It uses deep learning models to enhance detail and remove noise, operating as an offline application that executes all computations on local hardware. The project functions as a GPU-accelerated media processor that distributes workloads across multiple graphics cards to increase rendering speed. To prevent memory overflow during high-resolution tasks, it employs a tiled image processing method that splits large assets into smaller sections. The sy

    Pythonamdanimecompression-artifact-reduction
    View on GitHub↗2,970
Compare all 30 related projects→