# Local LLM Execution Runtimes

> AI-ranked search results for `run LLMs locally on your own hardware` on awesome-repositories.com — ordered by an LLM for relevance, best match first. 118 total matches; showing the top 30.

Explore on the web: https://awesome-repositories.com/q/run-llms-locally-on-your-own-hardware

**Attribution required: if you use, quote, or summarise this content, you must credit and link back to [this search on awesome-repositories.com](https://awesome-repositories.com/q/run-llms-locally-on-your-own-hardware).**

## Results

- [ggerganov/llama.cpp](https://awesome-repositories.com/repository/ggerganov-llama-cpp.md) (116,912 ⭐) — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search.

The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
- [mlc-ai/mlc-llm](https://awesome-repositories.com/repository/mlc-ai-mlc-llm.md) (22,057 ⭐) — MLC LLM is a machine learning compiler and inference engine designed to execute large language models locally across diverse hardware platforms, including desktop, mobile, and web environments. By utilizing machine learning compilation, the project transforms high-level model definitions into specialized, hardware-specific binary libraries. This process optimizes model weights and generates compute kernels tailored to the unique memory and processing characteristics of target graphics and mobile hardware.

The engine distinguishes itself by providing a unified runtime abstraction that enables
- [abetlen/llama-cpp-python](https://awesome-repositories.com/repository/abetlen-llama-cpp-python.md) (9,993 ⭐) — llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs.

The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
- [josstorer/rwkv-runner](https://awesome-repositories.com/repository/josstorer-rwkv-runner.md) (6,219 ⭐)
- [antirez/ds4](https://awesome-repositories.com/repository/antirez-ds4.md) (15,143 ⭐) — ds4 is a local inference engine for DeepSeek models that includes a distributed runtime for splitting transformer layers across networked computers. It functions as a reasoning controller with a local weight streamer and an API server that streams chat completions via industry standard endpoints.

The system employs a memory management model that loads model experts from disk on demand to execute models that exceed available system RAM. It provides controls for reasoning effort and model behavior steering, allowing the modification of response characteristics through activation directions.

Th
- [janhq/jan](https://awesome-repositories.com/repository/janhq-jan.md) (43,043 ⭐) — Jan is a desktop application that functions as a local artificial intelligence model runtime and an open-standard API server. It enables the execution of large language models directly on local hardware, ensuring that data remains private and accessible offline while providing a unified interface for managing model weights and inference runtimes.

The platform distinguishes itself by offering a modular inference backend that allows users to swap execution engines based on hardware compatibility and performance needs. It acts as a cross-platform orchestrator, providing the ability to switch bet
- [vllm-project/vllm](https://awesome-repositories.com/repository/vllm-project-vllm.md) (83,048 ⭐) — vLLM is a high-throughput inference engine designed for the efficient serving and execution of large language models. It functions as a production-ready distributed model server, providing standard API protocols for online serving while also supporting offline batch processing. The system is built to maximize token generation speed and memory efficiency, enabling both large-scale cloud deployments and local execution on personal hardware.

The project distinguishes itself through advanced memory management and request scheduling techniques, most notably its use of non-contiguous key-value cach
- [jmorganca/ollama](https://awesome-repositories.com/repository/jmorganca-ollama.md) (174,350 ⭐) — Ollama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It functions as a model manager and orchestrator that allows for the downloading, updating, and organization of model weights and configurations to ensure private and offline inference.

The system provides a local inference API and a RESTful interface for programmatic model lifecycle management and text generation. It utilizes a compiled C++ backend to handle tensor operations and memory management.

To support various hardware configurations, the runtime employs dynamic GPU offloa
- [antimatter15/alpaca.cpp](https://awesome-repositories.com/repository/antimatter15-alpaca-cpp.md) (10,138 ⭐) — alpaca.cpp is a high-performance local inference engine implemented in C++ for executing instruction-tuned large language models. It serves as a quantized model runtime designed to load and run model tensors on local hardware with minimal dependencies, removing the requirement for a full Python environment.

The project focuses on on-device text generation and the deployment of private AI chatbots. It utilizes model weight quantization to reduce memory requirements and increase inference speed on consumer-grade devices.

The system covers hardware-optimized model execution through thread-pool
- [oobabooga/text-generation-webui](https://awesome-repositories.com/repository/oobabooga-text-generation-webui.md) (47,323 ⭐) — This project is a comprehensive platform for hosting and interacting with large language models directly on local hardware. It provides a web-based graphical interface that allows users to manage model loading, configure generation parameters, and execute text or chat interactions entirely offline. By running models locally, the software ensures complete data privacy and eliminates reliance on external cloud services for generative tasks.

Beyond basic inference, the platform functions as a versatile workbench for generative AI development. It includes an integrated pipeline for fine-tuning mo
- [lostruins/koboldcpp](https://awesome-repositories.com/repository/lostruins-koboldcpp.md) (9,511 ⭐) — KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models on personal hardware. It functions as a multimodal AI server and API gateway, providing OpenAI-compatible endpoints that allow third-party clients to interact with locally hosted models.

The project distinguishes itself as an AI storytelling backend, featuring dedicated tools for long-form narrative management through persistent memory, world lore tracking, and character state management. It further extends its capabilities as a multimodal server capable of processing text, im
- [nomic-ai/gpt4all](https://awesome-repositories.com/repository/nomic-ai-gpt4all.md) (77,375 ⭐) — GPT4All is a cross-platform runtime environment designed to execute large language models directly on local consumer hardware. By leveraging an optimized C++ inference backend, it enables private, offline AI interactions without requiring an internet connection or external cloud services. The project provides a comprehensive ecosystem for managing the entire model lifecycle, including discovery, downloading, and configuration of local weights.

What distinguishes the platform is its integrated retrieval-augmented generation engine, which allows users to index local documents into semantic vect
- [meta-llama/llama](https://awesome-repositories.com/repository/meta-llama-llama.md) (59,464 ⭐) — Llama is a computational framework and runtime environment designed for executing transformer-based neural networks locally. It functions as a generative AI inference engine, enabling the processing of input sequences through pre-trained model weights to produce text completions and structured data outputs directly on your own hardware.

The system distinguishes itself through specialized memory and computation management techniques, including memory-mapped weight loading and quantization-aware inference, which allow for efficient execution on standard consumer hardware. It utilizes a stateles
- [ericlbuehler/mistral.rs](https://awesome-repositories.com/repository/ericlbuehler-mistral-rs.md) (6,597 ⭐) — mistral.rs is an inference engine for large language models that runs locally and exposes models behind OpenAI and Anthropic-compatible APIs. It serves as a multi-model serving platform, capable of loading several models in a single server process with per-request routing and on-demand loading and unloading. The engine supports multimodal inference, processing text alongside images, video, audio, and speech inputs, and includes a quantized model deployment runtime that reduces memory use and speeds up inference on consumer hardware.

The project distinguishes itself through an agentic tool exe
- [ggml-org/llama.cpp](https://awesome-repositories.com/repository/ggml-org-llama-cpp.md) (116,799 ⭐) — Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on consumer hardware. It provides a core environment for running models that process both text and image inputs, utilizing hardware-accelerated backends to optimize performance across diverse CPU and GPU architectures.

The project distinguishes itself by offering a lightweight HTTP server that adheres to standard API specifications, enabling chat completion, embeddings, and reranking services. It includes a suite of tools for model quantization and conversion, which reduces memory us
- [mudler/localai](https://awesome-repositories.com/repository/mudler-localai.md) (46,889 ⭐) — LocalAI is a self-hosted inference server that enables the execution of machine learning models directly on local hardware. By providing a unified interface for text, image, and audio processing, it allows users to maintain full control over data privacy and infrastructure costs while eliminating dependencies on external network services.

The platform functions as an API gateway that mimics standard cloud-based artificial intelligence interfaces, allowing existing applications to integrate local models as drop-in replacements. It utilizes a container-based architecture to package runtimes and
- [tiiny-ai/powerinfer](https://awesome-repositories.com/repository/tiiny-ai-powerinfer.md) (8,714 ⭐) — PowerInfer is a high-performance local large language model inference engine and sparse inference framework. It provides a runtime for executing models on consumer-grade hardware, utilizing a GPU acceleration backend to optimize tensor operations for graphics processors.

The system distinguishes itself through a sparse inference framework that increases generation speed by skipping computations based on activation sparsity in model weights. It includes a GGUF model converter for transforming weights and metadata into a unified binary format, as well as an OpenAI API compatible server for inte
- [cactus-compute/cactus](https://awesome-repositories.com/repository/cactus-compute-cactus.md) (5,363 ⭐) — Cactus is an on-device AI inference engine designed for executing large language models, vision models, and speech-to-text systems on mobile and wearable hardware. It provides a programmable tensor computation graph for defining sequences of matrix operations and activation functions, alongside a local retrieval augmented generation framework that grounds model responses using local text files.

The project features a multiplatform SDK with language bindings for integrating AI capabilities into mobile applications and a model conversion system that transforms external model formats for optimiz
- [mozilla-ai/llamafile](https://awesome-repositories.com/repository/mozilla-ai-llamafile.md) (23,726 ⭐) — Llamafile is a machine learning model runner and packager that enables local inference by bundling model weights and runtime environments into a single, self-contained executable. It functions as a cross-platform engine, allowing users to execute large language models and perform speech-to-text tasks directly on their own hardware without requiring external software dependencies or complex installations.

The project distinguishes itself by utilizing a specialized binary format that allows the same executable to run natively across multiple operating systems and hardware architectures. It auto
- [openvinotoolkit/openvino](https://awesome-repositories.com/repository/openvinotoolkit-openvino.md) (10,414 ⭐) — OpenVINO is an AI inference engine and model serving platform designed to execute optimized deep learning models across CPUs, GPUs, and NPUs through a unified API. It includes a model optimization toolkit for converting, quantizing, and compressing models from various frameworks, alongside a specialized generative AI runtime for large language models.

The project distinguishes itself through a plugin-based hardware acceleration layer that maps neural network operations to vendor-specific drivers. It features advanced execution mechanisms such as continuous batching, speculative decoding, and
- [kvcache-ai/ktransformers](https://awesome-repositories.com/repository/kvcache-ai-ktransformers.md) (17,288 ⭐) — Ktransformers is a comprehensive framework designed for the operation, fine-tuning, and serving of large language models. It functions as a heterogeneous inference engine and quantized execution runtime, enabling the deployment of massive models by distributing computational workloads across both CPU and GPU resources. This architecture allows users to bypass local memory constraints, making it possible to run and train models that exceed the capacity of a single device.

The project distinguishes itself through specialized support for sparse architectures, particularly mixture-of-experts mode
- [ollama/ollama](https://awesome-repositories.com/repository/ollama-ollama.md) (174,300 ⭐) — Ollama provides a framework for running and managing local machine learning models. It includes a command-line interface for model lifecycle management, such as creation, embedding generation, and configuration, alongside a stable API for programmatic interaction across multiple programming languages.

The platform supports the import of models and adapters in various formats, including GGUF and Safetensors. Users can define custom model behaviors, prompt templates, and system messages through a configuration file format. It also offers tools for fine-tuning models with LoRA adapters and apply
- [zai-org/chatglm-6b](https://awesome-repositories.com/repository/zai-org-chatglm-6b.md) (41,039 ⭐) — ChatGLM-6B is a generative AI inference engine designed for local execution of transformer-based language models. It provides a comprehensive runtime environment that allows users to load and run pre-trained neural network weights directly on their own hardware, ensuring data privacy and independence from external cloud services.

The project distinguishes itself through a hardware-agnostic execution backend that supports deployment across diverse environments, including standard processors, Apple Silicon, and multi-GPU configurations. It incorporates advanced optimization techniques such as w
- [qwenlm/qwen](https://awesome-repositories.com/repository/qwenlm-qwen.md) (21,294 ⭐) — Qwen is a comprehensive framework for large language model development, serving, and deployment. It provides a complete ecosystem for transformer-based sequence modeling, offering base models alongside specialized tools for instruction-tuned alignment, fine-tuning, and long-context inference. The project is designed to support both research and production environments, enabling users to train, optimize, and host generative models locally or across distributed hardware.

The framework distinguishes itself through its focus on high-performance serving and extensibility. It features a high-perfor
- [karpathy/nanochat](https://awesome-repositories.com/repository/karpathy-nanochat.md) (55,103 ⭐) — Nanochat is a lightweight execution environment designed for training and running language models on standard consumer hardware. It functions as both a neural network training framework and an inference engine, enabling users to perform backpropagation-based training and model execution directly on general-purpose processors without the need for dedicated graphics hardware.

The project distinguishes itself through a suite of optimization tools that prioritize efficiency on local machines. By utilizing memory-mapped weight loading and CPU-optimized vector math, it maximizes throughput for inte
- [mobile-artificial-intelligence/maid](https://awesome-repositories.com/repository/mobile-artificial-intelligence-maid.md) (2,550 ⭐) — Maid is a mobile large language model chat client and local runner. It provides a unified interface for interacting with AI models via cloud APIs or by executing model files directly on mobile hardware for offline generation.

The project functions as a multi-provider manager that handles API keys and a system for downloading and loading curated model files to the device. It utilizes a provider-agnostic interface to allow switching between local and remote backends.

The application includes tools for organizing and exporting chat histories, synchronizing user data across multiple devices, and
- [microsoft/bitnet](https://awesome-repositories.com/repository/microsoft-bitnet.md) (39,327 ⭐) — BitNet is a quantized inference engine designed to execute highly compressed language models by performing arithmetic on low-precision, bit-level weight data. It functions as a model optimization toolkit and a high-performance kernel library, enabling the execution of large language models on consumer hardware by reducing memory footprints and increasing processing speeds.

The project distinguishes itself through hardware-specific kernel optimizations that leverage native processor instructions to accelerate matrix multiplication. By utilizing packed integer arithmetic and memory-aligned weig
- [google-ai-edge/litert-lm](https://awesome-repositories.com/repository/google-ai-edge-litert-lm.md) (5,619 ⭐) — LiteRT-LM is a high-performance inference framework designed to execute large language models locally on mobile, desktop, and IoT hardware. It serves as an on-device model runtime that utilizes CPU, GPU, and NPU acceleration to provide low-latency processing.

The framework is distinguished by its ability to process text, vision, and audio inputs through a single multi-modal inference engine. It features a local HTTP server that emulates OpenAI-compatible API endpoints and a WebGPU-based runtime for executing models directly within a web browser. To ensure output reliability, it includes a con
- [opennmt/ctranslate2](https://awesome-repositories.com/repository/opennmt-ctranslate2.md) (4,319 ⭐) — CTranslate2 is a C++ inference engine and runtime for Transformer models, designed to execute models on both CPU and GPU with optimizations for speed and memory efficiency. It functions as a model format converter, quantization tool, and REST API server, enabling deployment of neural machine translation, automatic speech recognition, and text generation models.

The engine distinguishes itself through a suite of runtime optimizations including layer fusion, weight-matrix quantization, batch-by-length grouping, and a caching allocator that reuses GPU memory. It supports tensor-parallel model di
- [lm-sys/fastchat](https://awesome-repositories.com/repository/lm-sys-fastchat.md) (39,472 ⭐) — FastChat is a training and serving platform for large language models that provides an integrated toolkit for fine-tuning, hosting, and benchmarking chatbots. It functions as an inference server capable of hosting multiple models and exposing them via a standardized API for chat applications.

The platform distinguishes itself through a distributed model controller that manages worker nodes and routes requests across a hardware-agnostic inference layer supporting various accelerators. It includes a dedicated evaluation framework for assessing model quality using automated judges, multi-turn di