What are the main features of google/gemma.cpp?

The main features of google/gemma.cpp are: C++ Inference Runtimes, Generative Text Inference, On-Device Inference, Model Loading, Model Checkpoint Converters, On-Device Models, Vision-Language Models, Gated Linear Recurrent Layers.

Gemma.cpp | Awesome Repos

gemma.cpp is a C++ inference engine for Gemma, PaliGemma, and Griffin language models, designed to run directly on-device without Python dependencies. It provides a self-contained runtime that loads quantized model weights and performs text generation on CPU or GPU, along with a model checkpoint converter that transforms PyTorch or Keras checkpoints into a compact binary format for fast loading.

The engine supports multiple model architectures, including the Griffin recurrent architecture with gated linear recurrent layers and sliding-window attention for efficient long-sequence handling, as well as vision-language fusion for processing images alongside text prompts. It features a grammar-constrained decoder that enforces user-defined acceptance rules to produce structured or formatted output, and a callback-based token generation system that yields each output token to a user-supplied function for custom streaming or early termination.

The project compiles into a standalone shared library artifact, enabling integration into external CMake projects via FetchContent for building custom inference frontends. It covers the full inference pipeline from checkpoint conversion and weight loading through tokenization, single-step forward passes, and token decoding, all within a lightweight runtime that operates without external services.

Features

C++ Inference Runtimes - "A library for running Gemma, PaliGemma, and Griffin language models directly in C++ with no Python dependencies."
Generative Text Inference - Load a Gemma model and generate text responses from user prompts in an interactive terminal session.
On-Device Inference - Running large language models locally on a CPU or GPU for text generation without cloud dependencies.
Model Loading - Load a pre-trained model from a binary weight file and prepare it for inference.
Model Checkpoint Converters - Transforming PyTorch or Keras model checkpoints into a binary format for fast loading in C++ applications.
On-Device Models - "A lightweight runtime that loads quantized model weights and runs text generation on CPU or GPU without external services."
Vision-Language Models - Loads a PaliGemma model that encodes an image and text prompt jointly, then generates descriptive or question-answering responses.
Gated Linear Recurrent Layers - Uses a gated linear recurrent layer with sliding-window attention to reduce memory and handle longer sequences efficiently.
Inference Step Executions - Perform one forward pass through the neural network for a single token, mutating the internal activations and key-value cache.
Grammar-Constrained Samplers - Forcing token generation to follow a user-defined grammar or acceptance rule for structured output.
Text Tokenizers - Convert a string prompt into a vector of token IDs using a loaded tokenizer model.
Callback-Based Generators - Accept a tokenized prompt and produce output tokens one at a time, calling a user-defined callback for each token.
Token Decoders - Convert a vector of token IDs back into a human-readable string.
Weight Serialization - Converts PyTorch or Keras checkpoints into a binary blob with a fixed memory layout for zero-deserialization loading at runtime.
Token Generation Callbacks - Yields each output token to a user-supplied callback, enabling custom streaming, filtering, or early termination.
In-Place Cache Mutation - Runs one neural network inference step per token, mutating internal activations and key-value cache in place.
Griffin Recurrent Model Runtimes - Loading and running Griffin-based recurrent models that use less memory and handle longer sequences efficiently.
Custom Frontend Definitions - Replace the built-in interactive interface with a custom application that calls the model's generation and tokenizer functions.
Custom Frontend Development - Building custom applications that call model generation and tokenizer functions instead of using the built-in interactive interface.
Custom Frontend SDKs - "A shared library and CMake integration for embedding Gemma model inference into external C++ applications."
Vision-Language Inference - Processing images alongside text prompts to generate descriptive or question-answering responses using a multimodal model.
CMake FetchContent Integration - Compiles the inference engine into a standalone shared library for linking into external CMake projects via FetchContent.
AI & Machine Learning - Lightweight inference engine for Google's Gemma models

pytorch/executorch

4,296Ver en GitHub

ExecuTorch is a lightweight C++ runtime for deploying PyTorch models on mobile, embedded, and edge hardware. It provides an ahead-of-time compilation pipeline that exports, quantizes, and lowers model graphs into compact serialized programs, then executes them through a minimal runtime with hardware acceleration and on-device large language model inference capabilities. The project distinguishes itself through a hardware accelerator delegate system that partitions model subgraphs and offloads computation to specialized backends including NPUs, GPUs, and DSPs from Apple, Arm, Intel, MediaTek,

microsoft/onnxruntime

19,347Ver en GitHub

This project is a cross-platform machine learning inference engine designed to execute pre-trained models across diverse operating systems and hardware environments. It functions as a standardized execution framework that manages the entire lifecycle of model inference, from loading and graph optimization to hardware-accelerated execution and generative sequence management. The runtime distinguishes itself through a highly modular architecture that decouples model logic from hardware-specific kernels. By utilizing an execution provider abstraction, it enables developers to offload computation

abetlen/llama-cpp-python

9,993Ver en GitHub

llama-cpp-python provides a Python interface for the llama.cpp library, enabling the execution of large language models with hardware acceleration. It functions as a GGUF model loader and a structured text generator capable of running inference servers and multimodal runtimes for processing both text and image inputs. The project distinguishes itself through a local inference server that exposes model capabilities via an OpenAI-compatible web API. It supports advanced execution techniques including speculative decoding, weight quantization, and layer-based GPU offloading to manage memory acro

Historial de estrellas

llama.cpp is a high-performance C++ inference engine and runtime for executing large language models locally across various hardware architectures. It provides the core components for local model execution, including a dedicated model quantizer for compressing weights into the GGUF format and a system for generating text embeddings for semantic search. The project distinguishes itself through specialized memory and execution optimizations, such as block-wise weight quantization to reduce memory footprints and memory-mapped model loading. It supports structured text generation by using formal

googlegemma.cpp

Gemma.cpp

Features