For an inference engine for running local LLMs, the first results are thudm/chatglm2-6b (ChatGLM2-6B is an open-weight bilingual LLM with a 32K-token context window, designed for local deployment with precision quantization and multi-GPU support—exactly the kind of downloadable, runnable model this search targets), qwenlm/qwen3 (Qwen3 is an open-weight transformer-based large language model with publicly available pre-trained weights, quantization support, and local inference tooling for running on consumer hardware, directly matching your request for downloadable and locally runnable models) and meta-llama/llama. qwenlm/qwen and deepseek-ai/deepseek-v3 round out the shortlist. Compare the match explanations and check the project documentation against your requirements.
High-performance software frameworks designed to execute large language models directly on your local hardware infrastructure.
ChatGLM2-6B is an open-weight large language model designed for natural language conversations and text generation in both English and Chinese. It functions as a bilingual chat model capable of processing and maintaining coherence across text sequences up to 32K tokens. The model is optimized for local deployment through precision quantization, which reduces memory requirements to allow execution on consumer-grade hardware. It supports distributing model weights across multiple graphics cards to handle parameters that exceed the memory of a single device. The project covers capabilities for
ChatGLM2-6B is an open-weight bilingual LLM with a 32K-token context window, designed for local deployment with precision quantization and multi-GPU support—exactly the kind of downloadable, runnable model this search targets.
Qwen3 is a transformer-based large language model designed as a generative AI foundation for understanding, reasoning, and generating human language. It functions as a comprehensive ecosystem for model training, fine-tuning, and production-ready inference, providing the underlying architecture and weights necessary to build diverse artificial intelligence applications. The project distinguishes itself through extensive support for model quantization and distributed inference, enabling efficient execution across a wide range of hardware from consumer-grade devices to scalable cloud infrastruct
Qwen3 is an open-weight transformer-based large language model with publicly available pre-trained weights, quantization support, and local inference tooling for running on consumer hardware, directly matching your request for downloadable and locally runnable models.
Llama is a computational framework and runtime environment designed for executing transformer-based neural networks locally. It functions as a generative AI inference engine, enabling the processing of input sequences through pre-trained model weights to produce text completions and structured data outputs directly on your own hardware. The system distinguishes itself through specialized memory and computation management techniques, including memory-mapped weight loading and quantization-aware inference, which allow for efficient execution on standard consumer hardware. It utilizes a stateles
The Llama repo provides the official inference engine and downloadable model weights for executing Meta’s Llama series locally, with built‑in quantization and CPU/GPU support—though the license is not fully permissive, and community fine‑tuned variants are not included directly, it squarely fits the search for an open‑weight LLM you can run on your own hardware.
Qwen is a comprehensive framework for large language model development, serving, and deployment. It provides a complete ecosystem for transformer-based sequence modeling, offering base models alongside specialized tools for instruction-tuned alignment, fine-tuning, and long-context inference. The project is designed to support both research and production environments, enabling users to train, optimize, and host generative models locally or across distributed hardware. The framework distinguishes itself through its focus on high-performance serving and extensibility. It features a high-perfor
Qwen is a series of open-weight large language models with publicly available pre-trained weights, along with a framework that provides local inference tooling, quantization support, and CPU/GPU deployment options—directly matching the need for models that can be downloaded and run on your own hardware.
DeepSeek-V3 is a large language model that provides comprehensive resources for model utilization, including technical specifications, pre-trained weights, and evaluation benchmarks. The project details the core transformer architecture, including parameter counts and multi-token prediction modules, while supporting native 8-bit floating-point quantization. The repository offers extensive support for local and distributed inference through integration with multiple frameworks and engines. It includes documentation for deploying the model across various hardware configurations, such as GPUs an
DeepSeek-V3 provides pre-trained weights, supports local inference with integrated frameworks, includes native quantization, and documents deployment on GPUs, making it a direct and comprehensive answer for running a large open-weight model on your own hardware.
ChatGLM-6B is a generative AI inference engine designed for local execution of transformer-based language models. It provides a comprehensive runtime environment that allows users to load and run pre-trained neural network weights directly on their own hardware, ensuring data privacy and independence from external cloud services. The project distinguishes itself through a hardware-agnostic execution backend that supports deployment across diverse environments, including standard processors, Apple Silicon, and multi-GPU configurations. It incorporates advanced optimization techniques such as w
ChatGLM-6B is an inference engine and official repository for a specific open-weight large language model, enabling local execution with CPU and GPU support and quantization, which directly meets the visitor's request for downloadable, runnable models.
LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by
LLaVA is a multimodal large language model with publicly available pre-trained weights that can be downloaded and run locally using its inference tooling, covering model size options, quantization support, and community fine-tuned variants—though it is specialized for vision-language tasks rather than being a general-purpose text-only LLM, it squarely fits the open-weight LLM category.
GGML is a machine learning tensor library and neural network engine written in C. It functions as a compute-focused runtime designed to execute transformer-based models and perform complex mathematical operations on multi-dimensional arrays directly on local consumer hardware. The library distinguishes itself by enabling local inference for large language models and edge machine learning deployment without reliance on external cloud infrastructure. It achieves this through a tensor-based computation graph that organizes operations for efficient execution and memory management, alongside stati
ggml is a high-performance tensor library and runtime for running large language models locally, but it is not a model with publicly available weights itself — you would use it to execute models like LLaMA or Mistral on your hardware, not download it as a pre-trained LLM.
MLC LLM is a machine learning compiler and inference engine designed to execute large language models locally across diverse hardware platforms, including desktop, mobile, and web environments. By utilizing machine learning compilation, the project transforms high-level model definitions into specialized, hardware-specific binary libraries. This process optimizes model weights and generates compute kernels tailored to the unique memory and processing characteristics of target graphics and mobile hardware. The engine distinguishes itself by providing a unified runtime abstraction that enables
MLC LLM is an inference engine and compiler for running large language models locally, not a repository of open-weight models themselves—you would use it to serve existing model weights rather than download a pre-trained model from this repo.
Ollama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It functions as a model manager and orchestrator that allows for the downloading, updating, and organization of model weights and configurations to ensure private and offline inference. The system provides a local inference API and a RESTful interface for programmatic model lifecycle management and text generation. It utilizes a compiled C++ backend to handle tensor operations and memory management. To support various hardware configurations, the runtime employs dynamic GPU offloa
Ollama is a runtime and model manager for running open-weight LLMs locally, but it is not itself a large language model with publicly available weights—it orchestrates other models rather than providing the pre-trained weights you are looking for.
AgenticSeek is a multi-agent orchestration system designed to decompose complex user objectives into granular, actionable tasks. By coordinating a team of specialized autonomous workers, the platform manages end-to-end workflows, ensuring that each component of a project is assigned to the most capable agent for execution. The system operates as a local-first runtime, executing all artificial intelligence models directly on user hardware to maintain data sovereignty and privacy. It integrates a browser automation engine for autonomous web research and interaction, alongside a sandboxed enviro
AgenticSeek is a multi-agent orchestration platform that runs local LLMs on your hardware, but it is not itself an open-weight model with downloadable weights—it's a tool for orchestrating models, not the model you'd download and run directly.
alpaca.cpp is a high-performance local inference engine implemented in C++ for executing instruction-tuned large language models. It serves as a quantized model runtime designed to load and run model tensors on local hardware with minimal dependencies, removing the requirement for a full Python environment. The project focuses on on-device text generation and the deployment of private AI chatbots. It utilizes model weight quantization to reduce memory requirements and increase inference speed on consumer-grade devices. The system covers hardware-optimized model execution through thread-pool
alpaca.cpp is a local inference engine for running instruction-tuned language models, offering quantization and CPU-optimized execution, but it does not itself include or distribute pre-trained model weights — it is a runtime tool, not an open-weight model repository.
| Repository | Stars | Language | License | Last push |
|---|---|---|---|---|
| thudm/chatglm2-6b | 15.6K | Python | NOASSERTION | |
| qwenlm/qwen3 | 27.3K | Python | — | |
| meta-llama/llama | 59.6K | Python | NOASSERTION | |
| qwenlm/qwen | 22K | Python | Apache-2.0 | |
| deepseek-ai/deepseek-v3 | 104.8K | Python | MIT | |
| zai-org/chatglm-6b | 40.9K | Python | Apache-2.0 | |
| haotian-liu/llava | 24.5K | Python | apache-2.0 | |
| ggml-org/ggml | 14K | C++ | mit | |
| mlc-ai/mlc-llm | 22.1K | Python | apache-2.0 | |
| jmorganca/ollama | 174.4K | Go | MIT |