awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com
turboderp avatar

turboderp/exllamav2

0
View on GitHub↗
4,553 stars·337 forks·Python·MIT·32 views

Exllamav2

exllamav2 is a high-performance inference library designed for running large language models locally on consumer-grade GPUs. It provides a GPU-accelerated runner and quantization tools to enable model execution without reliance on cloud-based computing services.

The project features a quantization utility that compresses models into mixed bitrates between two and eight bits to reduce video RAM requirements. It distinguishes itself through a batched text generator that handles grouped requests and deduplicates cache data to increase throughput.

The library covers a broad capability surface including asynchronous token streaming for real-time output, custom GPU kernel execution for linear algebra operations, and local memory mapping for low-latency access to model weights.

Features

  • Local Inference Engines - Acts as a runtime optimized for executing large language models on consumer-grade hardware.
  • Batch Inference Engines - Provides a system that processes multiple input requests simultaneously to increase text generation throughput.
  • Batched Response Generation - Implements techniques for producing multiple model completions from a single prompt to increase throughput via batching.
  • GPU-Accelerated Inference - Implements hardware acceleration techniques specifically applied to the inference phase of models on GPUs.

AI search

Explore more awesome repositories

Describe what you need in plain English — the AI ranks thousands of curated open-source projects by relevance.

Start searching with AI
  • Inference Libraries - Provides a high-performance library to load and execute large language models on local hardware.
  • High-Throughput Text Inference - Delivers optimized model execution for high-volume text generation with low latency.
  • Mixed-Precision Quantization - Compresses model weights into varying bit-widths between two and eight bits to reduce memory footprint.
  • Model Quantization - Uses techniques to reduce the precision of model weights to decrease memory footprint and accelerate inference.
  • LLM Quantization Frameworks - Ships a toolkit designed to implement and manage the quantization of large language model weights.
  • Inference Batching - Groups multiple model inference requests into a single hardware execution pass to maximize GPU throughput.
  • CUDA Compute Kernels - Employs custom CUDA kernels written in C++ to parallelize heavy mathematical computations for LLM inference.
  • KV Cache Management - Optimizes inference efficiency by storing and reusing key-value pairs across multiple sequences in a batch.
  • Memory-Mapped Weight Loaders - Maps model weight files directly into process memory to reduce RAM usage and improve load times.
  • Incremental Inference Streaming - Provides a mechanism to send generated tokens to clients incrementally for real-time output.
  • Token Stream Generators - Uses a generator interface to output each generated token immediately for real-time incremental display.
  • Token Streamers - Provides components that deliver generated tokens incrementally to the user interface using asynchronous operations.
  • Real-Time Text Streaming - Implements low-latency processing of continuous token streams for real-time AI application output.
  • Consumer GPU Optimizations - Provides serving techniques tailored for running large models on consumer-grade hardware via precision reduction.
  • Inference Engines - High-speed inference library for modern consumer-grade GPUs.
  • Model Quantization - Fast inference library optimized for low-bitwidth model quantization.
  • Star history

    Star history chart for turboderp/exllamav2Star history chart for turboderp/exllamav2

    How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.

    Frequently asked questions

    What does turboderp/exllamav2 do?

    exllamav2 is a high-performance inference library designed for running large language models locally on consumer-grade GPUs. It provides a GPU-accelerated runner and quantization tools to enable model execution without reliance on cloud-based computing services.

    What are the main features of turboderp/exllamav2?

    The main features of turboderp/exllamav2 are: Local Inference Engines, Batch Inference Engines, Batched Response Generation, GPU-Accelerated Inference, Inference Libraries, High-Throughput Text Inference, Mixed-Precision Quantization, Model Quantization.

    Which projects share features with turboderp/exllamav2?

    Projects with overlapping indexed features include: turboderp-org/exllamav2 — exllamav2 is a high-performance inference engine and framework for executing large language models locally on… google-ai-edge/litert — LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT… kserve/kserve — KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference… predibase/lorax — Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It… ericlbuehler/mistral.rs — mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and… pytorch/serve — This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production…

    Projects sharing features with Exllamav2

    These projects share indexed features with Exllamav2. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.
    • turboderp-org/exllamav2turboderp-org avatar

      turboderp-org/exllamav2

      4,651View on GitHub↗

      exllamav2 is a high-performance inference engine and framework for executing large language models locally on consumer-class GPUs. It provides a complete system for local model deployment, including a specialized inference engine and tools for model quantization. The project features a multi-GPU inference framework that distributes workloads across multiple graphics cards to run models that exceed the memory capacity of a single device. It includes a GPU model quantizer capable of converting models into mixed-precision formats between 2 and 8 bits to balance memory usage and accuracy. The en

      Python
      View on GitHub↗4,651
    • google-ai-edge/litertgoogle-ai-edge avatar

      google-ai-edge/LiteRT

      3,577View on GitHub↗

      LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT hardware. It consists of an inference engine and a specialized environment for running quantized large language and diffusion models locally on edge hardware. The system includes an ahead-of-time model compiler that translates models into hardware-specific bytecode to reduce startup latency and memory overhead. It provides a unified interface for Neural Processing Units with automatic fallback routing to CPUs or GPUs when specific subgraph support is unavailable. An edge model conve

      C++
      View on GitHub↗3,577
    • kserve/kservekserve avatar

      kserve/kserve

      5,576View on GitHub↗

      KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere

      Go
      View on GitHub↗5,576
    • ericlbuehler/mistral.rsEricLBuehler avatar

      EricLBuehler/mistral.rs

      6,597View on GitHub↗

      mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe

      Rustllmrustuqff
      View on GitHub↗6,597
    Compare all 30 related projects→