google/gemma.cpp
Gemma.cpp
gemma.cpp is a C++ inference engine for Gemma, PaliGemma, and Griffin language models, designed to run directly on-device without Python dependencies. It provides a self-contained runtime that loads quantized model weights and performs text generation on CPU or GPU, along with a model checkpoint converter that transforms PyTorch or Keras checkpoints into a compact binary format for fast loading.
The engine supports multiple model architectures, including the Griffin recurrent architecture with gated linear recurrent layers and sliding-window attention for efficient long-sequence handling, as well as vision-language fusion for processing images alongside text prompts. It features a grammar-constrained decoder that enforces user-defined acceptance rules to produce structured or formatted output, and a callback-based token generation system that yields each output token to a user-supplied function for custom streaming or early termination.
The project compiles into a standalone shared library artifact, enabling integration into external CMake projects via FetchContent for building custom inference frontends. It covers the full inference pipeline from checkpoint conversion and weight loading through tokenization, single-step forward passes, and token decoding, all within a lightweight runtime that operates without external services.
Features
- C++ Inference Runtimes - "A library for running Gemma, PaliGemma, and Griffin language models directly in C++ with no Python dependencies."
- Generative Text Inference - Load a Gemma model and generate text responses from user prompts in an interactive terminal session.
- On-Device Inference - Running large language models locally on a CPU or GPU for text generation without cloud dependencies.
- Model Loading - Load a pre-trained model from a binary weight file and prepare it for inference.
- Model Checkpoint Converters - Transforming PyTorch or Keras model checkpoints into a binary format for fast loading in C++ applications.
- On-Device Models - "A lightweight runtime that loads quantized model weights and runs text generation on CPU or GPU without external services."
- Vision-Language Models - Loads a PaliGemma model that encodes an image and text prompt jointly, then generates descriptive or question-answering responses.
- Gated Linear Recurrent Layers - Uses a gated linear recurrent layer with sliding-window attention to reduce memory and handle longer sequences efficiently.
- Inference Step Executions - Perform one forward pass through the neural network for a single token, mutating the internal activations and key-value cache.
- Grammar-Constrained Samplers - Forcing token generation to follow a user-defined grammar or acceptance rule for structured output.
- Text Tokenizers - Convert a string prompt into a vector of token IDs using a loaded tokenizer model.
- Callback-Based Generators - Accept a tokenized prompt and produce output tokens one at a time, calling a user-defined callback for each token.
- Token Decoders - Convert a vector of token IDs back into a human-readable string.
- Weight Serialization - Converts PyTorch or Keras checkpoints into a binary blob with a fixed memory layout for zero-deserialization loading at runtime.
- Token Generation Callbacks - Yields each output token to a user-supplied callback, enabling custom streaming, filtering, or early termination.
- In-Place Cache Mutation - Runs one neural network inference step per token, mutating internal activations and key-value cache in place.
- Griffin Recurrent Model Runtimes - Loading and running Griffin-based recurrent models that use less memory and handle longer sequences efficiently.
- Custom Frontend Definitions - Replace the built-in interactive interface with a custom application that calls the model's generation and tokenizer functions.
- Custom Frontend Development - Building custom applications that call model generation and tokenizer functions instead of using the built-in interactive interface.
- Custom Frontend SDKs - "A shared library and CMake integration for embedding Gemma model inference into external C++ applications."
- Vision-Language Inference - Processing images alongside text prompts to generate descriptive or question-answering responses using a multimodal model.
- CMake FetchContent Integration - Compiles the inference engine into a standalone shared library for linking into external CMake projects via FetchContent.
- AI & Machine Learning - Lightweight inference engine for Google's Gemma models
