CPU GGUF Model Inference Engines
For an engine for running quantized models on CPU, the strongest matches are ggml-org/llama.cpp (Llama), ggerganov/llama.cpp (llama) and nomic-ai/gpt4all (GPT4All is a cross-platform, CPU-friendly inference engine that runs). oobabooga/text-generation-webui and abetlen/llama-cpp-python round out the shortlist. Each is ranked by relevance to your query, popularity and recent activity.
High-performance software libraries and runtimes designed to execute quantized GGUF machine learning models on CPUs.
CPU GGUF Model Inference Engines
Find the best repos with AI.We'll search the best matching repositories with AI.
- ggml-org/llama.cpp
ggml-org/llama.cpp
116,799View on GitHubLlama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on consumer hardware. It provides a core environment for running models that process both text and image inputs, utilizing hardware-accelerated backends to optimize performance across diverse CPU and GPU architectures. The project distinguishes itself by offering a lightweight HTTP server that adheres to standard API specifications, enabling chat completion, embeddings, and reranking services. It includes a suite of tools for model quantization and conversion, which reduces memory us
Llama.cpp is the reference inference engine for running quantized LLMs in GGUF format on CPUs, covering all requested features like CLI, cross-platform support, and CPU optimizations (AVX, NEON).
C++Hardware Abstraction LayersText-Only Inference EnginesMultimodal Inference Engines - ggerganov/llama.cpp
ggerganov/llama.cpp
116,912View on GitHubllama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
llama.cpp is the leading CPU-focused inference engine for GGUF models, with built-in quantisation, memory-mapped loading, and optimisations for ARM and x86 CPUs — it matches every required feature, making it the definitive answer for running quantised LLMs locally without a GPU.
C++Local Inference EnginesC++ Inference RuntimesCross-Platform Inference Frameworks - nomic-ai/gpt4all
nomic-ai/gpt4all
77,375View on GitHubGPT4All is a cross-platform runtime environment designed to execute large language models directly on local consumer hardware. By leveraging an optimized C++ inference backend, it enables private, offline AI interactions without requiring an internet connection or external cloud services. The project provides a comprehensive ecosystem for managing the entire model lifecycle, including discovery, downloading, and configuration of local weights. What distinguishes the platform is its integrated retrieval-augmented generation engine, which allows users to index local documents into semantic vect
GPT4All is a cross-platform, CPU-friendly inference engine that runs quantized LLMs locally using GGUF models, directly matching the request for a tool to execute GGUF models on consumer hardware without a GPU.
C++C++ Inference BackendsLanguage Model OrchestrationLocal AI Inference - oobabooga/text-generation-webui
oobabooga/text-generation-webui
47,323View on GitHubThis project is a comprehensive platform for hosting and interacting with large language models directly on local hardware. It provides a web-based graphical interface that allows users to manage model loading, configure generation parameters, and execute text or chat interactions entirely offline. By running models locally, the software ensures complete data privacy and eliminates reliance on external cloud services for generative tasks. Beyond basic inference, the platform functions as a versatile workbench for generative AI development. It includes an integrated pipeline for fine-tuning mo
oobabooga/text-generation-webui gives you a full web-based interface for loading and interacting with GGUF models locally, including on CPU, but it is a broader LLM workbench rather than a focused CPU-only inference engine — it supports multiple backends and GPU acceleration when available, which makes it less specialised than the lean CPU-first tool this search asks for.
PythonLocal Inference EnginesLocal Model RuntimesModel Serving APIs - abetlen/llama-cpp-python
abetlen/llama-cpp-python
9,993View on GitHubllama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
llama-cpp-python wraps the CPU-first llama.cpp engine, directly supporting GGUF quantized models, memory-efficient mmap loading, and AVX/NEON optimizations without requiring a GPU; while it emphasizes a Python API and inference server over a standalone command-line tool, it squarely delivers the CPU-based GGUF inference this search needs.
PythonLLM Python BindingsChat Completion ServicesEmbedding Generators - jmorganca/ollama
jmorganca/ollama
174,350View on GitHubOllama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It functions as a model manager and orchestrator that allows for the downloading, updating, and organization of model weights and configurations to ensure private and offline inference. The system provides a local inference API and a RESTful interface for programmatic model lifecycle management and text generation. It utilizes a compiled C++ backend to handle tensor operations and memory management. To support various hardware configurations, the runtime employs dynamic GPU offloa
Ollama is a cross-platform runtime for running large language models locally, supporting CPU inference and quantized models through its C++ backend, which fits the search for a CPU-based GGUF inference engine, though its explicit GGUF support and CPU-only focus are not highlighted in the provided evidence.
GoLocal Model ExecutionAI Integration APIsLanguage Model Orchestration - michael-a-kuykendall/shimmy
Michael-A-Kuykendall/shimmy
5,428View on GitHubShimmy is a local large language model inference engine and server that loads and serves GGUF formatted weights. It is distributed as a single binary runtime written in Rust, providing a standalone environment for running models without external runtime dependencies. The project utilizes WebGPU for hardware acceleration, allowing model compute kernels to execute across diverse graphics hardware through a standardized interface. It features a local server that implements an OpenAI-compatible API layer, enabling applications to interface with local models via standardized REST endpoints. Memor
Shimmy is a GGUF model inference engine, but it relies on WebGPU for hardware acceleration (including GPUs) rather than being CPU-only, which does not match the specific requirement for a CPU-based inference engine.
RustGGUF Inference EnginesGGUF Model LoadersGGUF Model Loaders - nsarrazin/serge
nsarrazin/serge
5,725View on GitHubA web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
Serge is a web interface and API server that runs on top of llama.cpp, so it enables CPU-based GGUF inference, but it is not itself the inference engine—the visitor who wants a direct, command-line tool for running GGUF models would be better served by the underlying llama.cpp project.
SvelteGGUF Model LoadersGGUF Model Loaders - microsoft/bitnet
microsoft/BitNet
39,327View on GitHubBitNet is a quantized inference engine designed to execute highly compressed language models by performing arithmetic on low-precision, bit-level weight data. It functions as a model optimization toolkit and a high-performance kernel library, enabling the execution of large language models on consumer hardware by reducing memory footprints and increasing processing speeds. The project distinguishes itself through hardware-specific kernel optimizations that leverage native processor instructions to accelerate matrix multiplication. By utilizing packed integer arithmetic and memory-aligned weig
BitNet is a quantized inference engine for compressed language models, but it focuses on its own bit-level format rather than supporting the GGUF format, so it doesn't directly match your search for a GGUF-focused CPU inference tool.
PythonQuantized Inference RuntimesKernel Optimizations - kyutai-labs/pocket-tts
kyutai-labs/pocket-tts
3,301View on GitHubPocket-tts is a text-to-speech server and neural speech synthesizer that converts written text into audible speech. It includes a CPU-optimized inference engine and a voice cloning tool capable of analyzing audio samples to reproduce specific speaker characteristics. The system differentiates itself through the use of dynamic int8 quantization to reduce memory usage and increase generation speed on processors. It supports real-time speech synthesis by streaming audio chunks incrementally and utilizes voice state caching to store processed embeddings as portable files, bypassing redundant proc
Pocket-tts is a CPU-optimized text-to-speech engine with dynamic quantization, but it generates speech rather than running general‑purpose LLMs in GGUF format, so it doesn’t serve the core capability you’re looking for.
PythonCPU Inference RuntimesCPU Inference Quantizers - hunyuan-promptenhancer/promptenhancer
Hunyuan-PromptEnhancer/PromptEnhancer
3,421View on GitHubPromptEnhancer is a GGUF prompt rewriting engine that loads quantized models to rewrite plain text prompts into clearer, structured versions while preserving the original subject, style, and layout. It is designed to refine image editing instructions by incorporating visual context from the input image, producing precise editing prompts. The tool operates through a structured prompt rewriting engine that combines editing instructions with visual context from images, embedding that context into the rewriting process for context-aware refinement. It runs inference with a minimal memory footprin
PromptEnhancer is a specialized prompt rewriting tool that happens to load quantized GGUF models for inference, but it is not a general-purpose inference engine for running any LLM on CPU—it's a narrow application built on top of that capability.
PythonQuantized Inference Runtimes - fminference/flexgen
FMInference/FlexGen
9,366View on GitHubFlexGen is an inference engine for large language models designed for high-throughput execution on single or multiple GPUs. It functions as a framework for managing model execution through a combination of memory offloading, weight compression, and pipeline orchestration. The system enables the execution of models that exceed available GPU memory by moving tensors and caches between GPU memory, system RAM, and disk storage. It utilizes 4-bit weight quantization to reduce the memory footprint of model parameters, allowing for increased batch processing capacity. The project covers distributed
FlexGen is designed for GPU-based high-throughput inference with memory offloading to CPU and disk, not as a CPU-only GGUF inference engine, so it does not match the search for a tool that runs quantized GGUF models solely on CPU.
PythonQuantized Model Runners
