Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on consumer hardware. It provides a core environment for running models that process both text and image inputs, utilizing hardware-accelerated backends to optimize performance across diverse CPU and GPU architectures. The project distinguishes itself by offering a lightweight HTTP server that adheres to standard API specifications, enabling chat completion, embeddings, and reranking services. It includes a suite of tools for model quantization and conversion, which reduces memory us
Jan is a desktop application that functions as a local artificial intelligence model runtime and an open-standard API server. It enables the execution of large language models directly on local hardware, ensuring that data remains private and accessible offline while providing a unified interface for managing model weights and inference runtimes. The platform distinguishes itself by offering a modular inference backend that allows users to swap execution engines based on hardware compatibility and performance needs. It acts as a cross-platform orchestrator, providing the ability to switch bet
llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
Ollama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It functions as a model manager and orchestrator that allows for the downloading, updating, and organization of model weights and configurations to ensure private and offline inference. The system provides a local inference API and a RESTful interface for programmatic model lifecycle management and text generation. It utilizes a compiled C++ backend to handle tensor operations and memory management. To support various hardware configurations, the runtime employs dynamic GPU offloa
The main features of janhq/cortex.cpp are: AI & Machine Learning, Inference Engines.
Open-source alternatives to janhq/cortex.cpp include: janhq/jan — Jan is a desktop application that functions as a local artificial intelligence model runtime and an open-standard API… langchain-ai/langchain — LangChain is an orchestration framework designed for building, managing, and deploying applications powered by large… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… ggml-org/llama.cpp — Llama.cpp is an inference engine designed for the local execution of text-based and multimodal language models on… jmorganca/ollama — Ollama is a cross-platform runtime for managing, serving, and executing large language models on local hardware. It… michael-a-kuykendall/shimmy — Shimmy is a local large language model inference engine and server that loads and serves GGUF formatted weights. It is…