awesome-repositories.com
Blog
MCP
awesome-repositories.com

Discover the best open-source repositories with AI-powered search.

ExploreCurated searchesOpen-source alternativesSelf-hosted softwareBlogSitemap
ProjectMCP serverAboutHow we rankPress
LegalPrivacyTerms
© 2026 Bringes Technology SRL·VAT RO45896025·hello@awesome-repositories.com

Local Open-Weight LLM Inference Engines

Ranking updated Jun 30, 2026

For an inference engine for running local LLMs, the first results are thudm/chatglm2-6b (ChatGLM2-6B is an open-weight bilingual LLM with a 32K-token context window, designed for local deployment with precision quantization and multi-GPU support—exactly the kind of downloadable, runnable model this search targets), qwenlm/qwen3 (Qwen3 is an open-weight transformer-based large language model with publicly available pre-trained weights, quantization support, and local inference tooling for running on consumer hardware, directly matching your request for downloadable and locally runnable models) and meta-llama/llama. qwenlm/qwen and deepseek-ai/deepseek-v3 round out the shortlist. Compare the match explanations and check the project documentation against your requirements.

High-performance software frameworks designed to execute large language models directly on your local hardware infrastructure.

Local Open-Weight LLM Inference Engines

Find the best repos with AI.We'll search the best matching repositories with AI.
  • thudm/chatglm2-6bTHUDM avatar

    THUDM/ChatGLM2-6B

    15,565View on GitHub↗

    ChatGLM2-6B is an open-weight large language model designed for natural language conversations and text generation in both English and Chinese. It functions as a bilingual chat model capable of processing and maintaining coherence across text sequences up to 32K tokens. The model is optimized for local deployment through precision quantization, which reduces memory requirements to allow execution on consumer-grade hardware. It supports distributing model weights across multiple graphics cards to handle parameters that exceed the memory of a single device. The project covers capabilities for

    ChatGLM2-6B is an open-weight bilingual LLM with a 32K-token context window, designed for local deployment with precision quantization and multi-GPU support—exactly the kind of downloadable, runnable model this search targets.

    PythonOpen-Weights ModelsPrecision Quantization
    View on GitHub↗15,565
  • qwenlm/qwen3QwenLM avatar

    QwenLM/Qwen3

    27,324View on GitHub↗

    Qwen3 is a transformer-based large language model designed as a generative AI foundation for understanding, reasoning, and generating human language. It functions as a comprehensive ecosystem for model training, fine-tuning, and production-ready inference, providing the underlying architecture and weights necessary to build diverse artificial intelligence applications. The project distinguishes itself through extensive support for model quantization and distributed inference, enabling efficient execution across a wide range of hardware from consumer-grade devices to scalable cloud infrastruct

    Qwen3 is an open-weight transformer-based large language model with publicly available pre-trained weights, quantization support, and local inference tooling for running on consumer hardware, directly matching your request for downloadable and locally runnable models.

    PythonLocal Inference EnginesModel Quantization
    View on GitHub↗27,324
  • meta-llama/llamameta-llama avatar

    meta-llama/llama

    59,620View on GitHub↗

    Llama is a computational framework and runtime environment designed for executing transformer-based neural networks locally. It functions as a generative AI inference engine, enabling the processing of input sequences through pre-trained model weights to produce text completions and structured data outputs directly on your own hardware. The system distinguishes itself through specialized memory and computation management techniques, including memory-mapped weight loading and quantization-aware inference, which allow for efficient execution on standard consumer hardware. It utilizes a stateles

    The Llama repo provides the official inference engine and downloadable model weights for executing Meta’s Llama series locally, with built‑in quantization and CPU/GPU support—though the license is not fully permissive, and community fine‑tuned variants are not included directly, it squarely fits the search for an open‑weight LLM you can run on your own hardware.

    PythonLocal Inference EnginesLocal Inference Runners
    View on GitHub↗59,620
  • qwenlm/qwenQwenLM avatar

    QwenLM/Qwen

    22,014View on GitHub↗

    Qwen is a comprehensive framework for large language model development, serving, and deployment. It provides a complete ecosystem for transformer-based sequence modeling, offering base models alongside specialized tools for instruction-tuned alignment, fine-tuning, and long-context inference. The project is designed to support both research and production environments, enabling users to train, optimize, and host generative models locally or across distributed hardware. The framework distinguishes itself through its focus on high-performance serving and extensibility. It features a high-perfor

    Qwen is a series of open-weight large language models with publicly available pre-trained weights, along with a framework that provides local inference tooling, quantization support, and CPU/GPU deployment options—directly matching the need for models that can be downloaded and run on your own hardware.

    PythonLocal AI InferenceModel Quantization
    View on GitHub↗22,014
  • deepseek-ai/deepseek-v3deepseek-ai avatar

    deepseek-ai/DeepSeek-V3

    104,757View on GitHub↗

    DeepSeek-V3 is a large language model that provides comprehensive resources for model utilization, including technical specifications, pre-trained weights, and evaluation benchmarks. The project details the core transformer architecture, including parameter counts and multi-token prediction modules, while supporting native 8-bit floating-point quantization. The repository offers extensive support for local and distributed inference through integration with multiple frameworks and engines. It includes documentation for deploying the model across various hardware configurations, such as GPUs an

    DeepSeek-V3 provides pre-trained weights, supports local inference with integrated frameworks, includes native quantization, and documents deployment on GPUs, making it a direct and comprehensive answer for running a large open-weight model on your own hardware.

    PythonModel WeightsInference FrameworksFrontier Models
    View on GitHub↗104,757
  • zai-org/chatglm-6bzai-org avatar

    zai-org/ChatGLM-6B

    40,863View on GitHub↗

    ChatGLM-6B is a generative AI inference engine designed for local execution of transformer-based language models. It provides a comprehensive runtime environment that allows users to load and run pre-trained neural network weights directly on their own hardware, ensuring data privacy and independence from external cloud services. The project distinguishes itself through a hardware-agnostic execution backend that supports deployment across diverse environments, including standard processors, Apple Silicon, and multi-GPU configurations. It incorporates advanced optimization techniques such as w

    ChatGLM-6B is an inference engine and official repository for a specific open-weight large language model, enabling local execution with CPU and GPU support and quantization, which directly meets the visitor's request for downloadable, runnable models.

    PythonLocal Inference EnginesModel Quantization
    View on GitHub↗40,863
  • haotian-liu/llavahaotian-liu avatar

    haotian-liu/LLaVA

    24,465View on GitHub↗

    LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

    LLaVA is a multimodal large language model with publicly available pre-trained weights that can be downloaded and run locally using its inference tooling, covering model size options, quantization support, and community fine-tuned variants—though it is specialized for vision-language tasks rather than being a general-purpose text-only LLM, it squarely fits the open-weight LLM category.

    PythonMultimodal Large Language ModelsVision-Language PipelinesVisual Instruction Tuning
    View on GitHub↗24,465
  • ggml-org/ggmlggml-org avatar

    ggml-org/ggml

    13,985View on GitHub↗

    GGML is a machine learning tensor library and neural network engine written in C. It functions as a compute-focused runtime designed to execute transformer-based models and perform complex mathematical operations on multi-dimensional arrays directly on local consumer hardware. The library distinguishes itself by enabling local inference for large language models and edge machine learning deployment without reliance on external cloud infrastructure. It achieves this through a tensor-based computation graph that organizes operations for efficient execution and memory management, alongside stati

    ggml is a high-performance tensor library and runtime for running large language models locally, but it is not a model with publicly available weights itself — you would use it to execute models like LLaMA or Mistral on your hardware, not download it as a pre-trained LLM.

    C++Local Inference EnginesLocal Language Model ExecutionPrecision Quantization
    View on GitHub↗13,985
  • mlc-ai/mlc-llmmlc-ai avatar

    mlc-ai/mlc-llm

    22,057View on GitHub↗

    MLC LLM is a machine learning compiler and inference engine designed to execute large language models locally across diverse hardware platforms, including desktop, mobile, and web environments. By utilizing machine learning compilation, the project transforms high-level model definitions into specialized, hardware-specific binary libraries. This process optimizes model weights and generates compute kernels tailored to the unique memory and processing characteristics of target graphics and mobile hardware. The engine distinguishes itself by providing a unified runtime abstraction that enables

    MLC LLM is an inference engine and compiler for running large language models locally, not a repository of open-weight models themselves—you would use it to serve existing model weights rather than download a pre-trained model from this repo.

    PythonLocal Inference EnginesLocal Language Model ExecutionModel Quantization
    View on GitHub↗22,057
  • jmorganca/ollamajmorganca avatar

    jmorganca/ollama

    174,350View on GitHub↗

    Ollama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It functions as a model manager and orchestrator that allows for the downloading, updating, and organization of model weights and configurations to ensure private and offline inference. The system provides a local inference API and a RESTful interface for programmatic model lifecycle management and text generation. It utilizes a compiled C++ backend to handle tensor operations and memory management. To support various hardware configurations, the runtime employs dynamic GPU offloa

    Ollama is a runtime and model manager for running open-weight LLMs locally, but it is not itself a large language model with publicly available weights—it orchestrates other models rather than providing the pre-trained weights you are looking for.

    GoLocal AI InferenceLocal Inference RuntimesPrecision Quantization
    View on GitHub↗174,350
  • fosowl/agenticseekFosowl avatar

    Fosowl/agenticSeek

    26,529View on GitHub↗

    AgenticSeek is a multi-agent orchestration system designed to decompose complex user objectives into granular, actionable tasks. By coordinating a team of specialized autonomous workers, the platform manages end-to-end workflows, ensuring that each component of a project is assigned to the most capable agent for execution. The system operates as a local-first runtime, executing all artificial intelligence models directly on user hardware to maintain data sovereignty and privacy. It integrates a browser automation engine for autonomous web research and interaction, alongside a sandboxed enviro

    AgenticSeek is a multi-agent orchestration platform that runs local LLMs on your hardware, but it is not itself an open-weight model with downloadable weights—it's a tool for orchestrating models, not the model you'd download and run directly.

    PythonLocal AI RuntimesLocal Inference EnginesLocal Model Runtimes
    View on GitHub↗26,529
  • antimatter15/alpaca.cppantimatter15 avatar

    antimatter15/alpaca.cpp

    10,138View on GitHub↗

    alpaca.cpp is a high-performance local inference engine implemented in C++ for executing instruction-tuned large language models. It serves as a quantized model runtime designed to load and run model tensors on local hardware with minimal dependencies, removing the requirement for a full Python environment. The project focuses on on-device text generation and the deployment of private AI chatbots. It utilizes model weight quantization to reduce memory requirements and increase inference speed on consumer-grade devices. The system covers hardware-optimized model execution through thread-pool

    alpaca.cpp is a local inference engine for running instruction-tuned language models, offering quantization and CPU-optimized execution, but it does not itself include or distribute pre-trained model weights — it is a runtime tool, not an open-weight model repository.

    CLocal Inference EnginesLocal Language Model ExecutionModel Quantization
    View on GitHub↗10,138
Compare the top 10 at a glance
RepositoryStarsLanguageLicenseLast push
thudm/chatglm2-6b15.6KPythonNOASSERTIONJun 27, 2024
qwenlm/qwen327.3KPython—Jan 9, 2026
meta-llama/llama59.6KPythonNOASSERTIONJan 26, 2025
qwenlm/qwen22KPythonApache-2.0Mar 5, 2026
deepseek-ai/deepseek-v3104.8KPythonMITAug 28, 2025
zai-org/chatglm-6b40.9KPythonApache-2.0Jun 27, 2024
haotian-liu/llava24.5KPythonapache-2.0Aug 12, 2024
ggml-org/ggml14KC++mitFeb 15, 2026
mlc-ai/mlc-llm22.1KPythonapache-2.0Feb 18, 2026
jmorganca/ollama174.4KGoMITJun 17, 2026

Related searches

  • a local runtime for executing LLMs
  • an open source engine for local LLMs
  • an open source platform for local LLMs
  • an open source model for local inference
  • an open source platform for local LLMs
  • LLM inference and serving
  • a self hosted platform for running LLMs
  • a desktop client for running local LLMs