What does google/gemma.cpp do?

gemma.cpp is a C++ inference engine for Gemma, PaliGemma, and Griffin language models, designed to run directly on-device without Python dependencies. It provides a self-contained runtime that loads quantized model weights and performs text generation on CPU or GPU, along with a model checkpoint converter that transforms PyTorch or Keras checkpoints into a compact binary format for fast loading.

What are the main features of google/gemma.cpp?

The main features of google/gemma.cpp are: C++ Inference Runtimes, Generative Text Inference, On-Device Inference, Model Loading, Model Checkpoint Converters, On-Device Models, Vision-Language Models, Gated Linear Recurrent Layers.

Which projects share features with google/gemma.cpp?

Projects with overlapping indexed features include: pytorch/executorch — ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It… microsoft/onnxruntime — This project is a cross-platform machine learning inference engine designed to execute pre-trained models across… abetlen/llama-cpp-python — llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language… ggerganov/llama.cpp — llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across… alibaba/mnn — MNN is a high-performance inference engine and framework designed for on-device machine learning. It provides a… lostruins/koboldcpp — KoboldCPP is a local large language model inference engine and GGUF model runner designed to execute quantized models…

Gemma.cpp - run Gemma models on-device in C++

Name: google/gemma.cpp
Author: google

Features

C++ Inference Runtimes - "A library for running Gemma, PaliGemma, and Griffin language models directly in C++ with no Python dependencies."
Generative Text Inference - Load a Gemma model and generate text responses from user prompts in an interactive terminal session.
On-Device Inference - Running large language models locally on a CPU or GPU for text generation without cloud dependencies.
Model Loading - Load a pre-trained model from a binary weight file and prepare it for inference.
Model Checkpoint Converters - Transforming PyTorch or Keras model checkpoints into a binary format for fast loading in C++ applications.
On-Device Models - "A lightweight runtime that loads quantized model weights and runs text generation on CPU or GPU without external services."
Vision-Language Models - Loads a PaliGemma model that encodes an image and text prompt jointly, then generates descriptive or question-answering responses.
Gated Linear Recurrent Layers - Uses a gated linear recurrent layer with sliding-window attention to reduce memory and handle longer sequences efficiently.
Inference Step Executions - Perform one forward pass through the neural network for a single token, mutating the internal activations and key-value cache.
Grammar-Constrained Samplers - Forcing token generation to follow a user-defined grammar or acceptance rule for structured output.
Text Tokenizers - Convert a string prompt into a vector of token IDs using a loaded tokenizer model.
Callback-Based Generators - Accept a tokenized prompt and produce output tokens one at a time, calling a user-defined callback for each token.
Token Decoders - Convert a vector of token IDs back into a human-readable string.
Weight Serialization - Converts PyTorch or Keras checkpoints into a binary blob with a fixed memory layout for zero-deserialization loading at runtime.
Token Generation Callbacks - Yields each output token to a user-supplied callback, enabling custom streaming, filtering, or early termination.
In-Place Cache Mutation - Runs one neural network inference step per token, mutating internal activations and key-value cache in place.
Griffin Recurrent Model Runtimes - Loading and running Griffin-based recurrent models that use less memory and handle longer sequences efficiently.
Custom Frontend Definitions - Replace the built-in interactive interface with a custom application that calls the model's generation and tokenizer functions.
Custom Frontend Development - Building custom applications that call model generation and tokenizer functions instead of using the built-in interactive interface.
Custom Frontend SDKs - "A shared library and CMake integration for embedding Gemma model inference into external C++ applications."
Vision-Language Inference - Processing images alongside text prompts to generate descriptive or question-answering responses using a multimodal model.
CMake FetchContent Integration - Compiles the inference engine into a standalone shared library for linking into external CMake projects via FetchContent.
AI & Machine Learning - Lightweight inference engine for Google's Gemma models

Projects sharing features with Gemma.cpp

These projects share indexed features with Gemma.cpp. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.

pytorch/executorch
pytorch/executorch
4,296View on GitHub
ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,
Pythondeep-learningembeddedgpu
View on GitHub4,296
microsoft/onnxruntime
microsoft/onnxruntime
19,347View on GitHub
This project is a cross-platform machine learning inference engine designed to execute pre-trained models across diverse operating systems and hardware environments. It functions as a standardized execution framework that manages the entire lifecycle of model inference, from loading and graph optimization to hardware-accelerated execution and generative sequence management. The runtime distinguishes itself through a highly modular architecture that decouples model logic from hardware-specific kernels. By utilizing an execution provider abstraction, it enables developers to offload computation
C++ai-frameworkdeep-learninghardware-acceleration
View on GitHub19,347
abetlen/llama-cpp-python
abetlen/llama-cpp-python
9,993View on GitHub
llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
Python
View on GitHub9,993
ggerganov/llama.cpp
ggerganov/llama.cpp
116,912View on GitHub
llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
C++
View on GitHub116,912

Compare all 30 related projects

Projects sharing features with Gemma.cpp

These projects share indexed features with Gemma.cpp. Shared tags can include platform or build tooling; verify the primary use case before treating a result as a replacement.

pytorch/executorch
pytorch/executorch
4,296View on GitHub
ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,
Pythondeep-learningembeddedgpu
View on GitHub4,296
microsoft/onnxruntime
microsoft/onnxruntime
19,347View on GitHub
This project is a cross-platform machine learning inference engine designed to execute pre-trained models across diverse operating systems and hardware environments. It functions as a standardized execution framework that manages the entire lifecycle of model inference, from loading and graph optimization to hardware-accelerated execution and generative sequence management. The runtime distinguishes itself through a highly modular architecture that decouples model logic from hardware-specific kernels. By utilizing an execution provider abstraction, it enables developers to offload computation
C++ai-frameworkdeep-learninghardware-acceleration
View on GitHub19,347
abetlen/llama-cpp-python
abetlen/llama-cpp-python
9,993View on GitHub
llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro
Python
View on GitHub9,993
ggerganov/llama.cpp
ggerganov/llama.cpp
116,912View on GitHub
llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal
C++
View on GitHub116,912

Compare all 30 related projects

googlegemma.cpp

Gemma.cpp

Features

Projects sharing features with Gemma.cpp

pytorch/executorch

microsoft/onnxruntime

abetlen/llama-cpp-python

ggerganov/llama.cpp

Frequently asked questions

Star history

Projects sharing features with Gemma.cpp

pytorch/executorch

microsoft/onnxruntime

abetlen/llama-cpp-python

ggerganov/llama.cpp

Frequently asked questions