How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
exllamav2 is a high-performance inference library designed for running large language models locally on consumer-grade GPUs. It provides a GPU-accelerated runner and quantization tools to enable model execution without reliance on cloud-based computing services.
The main features of turboderp/exllamav2 are: Local Inference Engines, Batch Inference Engines, Batched Response Generation, GPU-Accelerated Inference, Inference Libraries, High-Throughput Text Inference, Mixed-Precision Quantization, Model Quantization.
Projects with overlapping indexed features include: turboderp-org/exllamav2 — exllamav2 is a high-performance inference engine and framework for executing large language models locally on… google-ai-edge/litert — LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT… kserve/kserve — KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference… predibase/lorax — Lorax is a GPU-accelerated inference server and multi-adapter engine designed for serving large language models. It… ericlbuehler/mistral.rs — mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and… pytorch/serve — This project is a PyTorch model serving framework designed to deploy and scale machine learning models in production…
exllamav2 is a high-performance inference engine and framework for executing large language models locally on consumer-class GPUs. It provides a complete system for local model deployment, including a specialized inference engine and tools for model quantization. The project features a multi-GPU inference framework that distributes workloads across multiple graphics cards to run models that exceed the memory capacity of a single device. It includes a GPU model quantizer capable of converting models into mixed-precision formats between 2 and 8 bits to balance memory usage and accuracy. The en
LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT hardware. It consists of an inference engine and a specialized environment for running quantized large language and diffusion models locally on edge hardware. The system includes an ahead-of-time model compiler that translates models into hardware-specific bytecode to reduce startup latency and memory overhead. It provides a unified interface for Neural Processing Units with automatic fallback routing to CPUs or GPUs when specific subgraph support is unavailable. An edge model conve
KServe is a Kubernetes-native platform for deploying and serving machine learning models as scalable inference services. It supports both generative AI models, including large language models, and traditional predictive models from frameworks such as TensorFlow, PyTorch, Scikit-Learn, XGBoost, and ONNX. The platform manages the full lifecycle of model deployments, including revision tracking, canary rollouts, A/B testing, and automatic rollbacks, and provides serverless scale-to-zero capabilities for cost-efficient resource management. KServe distinguishes itself through a standardized infere
mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware. The project distinguishes itself through an agentic tool exe