google-ai-edge/LiteRT-LM

Features

• On-Device Inference - Executes large language models locally on Linux, macOS, Windows, and Raspberry Pi using local hardware.
• Local Model Execution - Provides a high-performance framework for running large language models locally on mobile, desktop, and IoT hardware.
• Edge AI Model Deployment - Optimizes and deploys generative AI models for high-performance execution on local CPU, GPU, and NPU hardware.
• On-Device Model Runtimes - Provides a high-performance runtime for executing large language models locally on mobile, desktop, and IoT hardware.
• Agent Tool Execution - Implements mechanisms for models to invoke external functions to automate complex agentic workflows.
• AI Conversation Managers - Manages AI chat sessions including system instructions, message history, and sampling parameters.
• Conversation Management Systems - Creates isolated conversation sessions with customizable system prompts to maintain user-specific context.
• Conversation State Management - Tracks interaction history and context between users and models to ensure dialogue continuity.
• Chat Model Text Generators - Generates text responses from user prompts using chat-specific configurations for local model inference.
• External Tool Integration - Allows the model to request the execution of client-side functions to interact with external data and APIs.
• Hardware Acceleration Backends - Implements hardware acceleration backends that route operations to CPU, GPU, and NPU for high-performance edge inference.
• Browser-Based Inference - Executes large language models directly in the web browser using hardware-accelerated WebGPU runtimes.
• Inference Frameworks - Provides a high-performance inference framework for executing large language models on mobile, desktop, and IoT hardware.
• Hardware Acceleration - Leverages GPU and NPU accelerators to maximize processing speed for on-device model operations.
• Multi-Modal Inference Engines - Provides a local runtime capable of processing text, vision, and audio inputs within a single model.
• Multi-Modal Inference Engines - Features a single execution engine capable of processing text, vision, and audio inputs simultaneously on-device.
• Edge Model Compilers - Converts large language models into specialized, quantized runtime files tailored for mobile, desktop, and IoT hardware.
• Model Inference Execution - Executes data through a processing pipeline to generate linguistic outputs via blocking or streaming calls.
• Multimodal Data Processing - Supports processing a combination of text, images, and audio data within a single inference engine.
• On-Device Inference Engines - Provides a high-performance runtime optimized for executing large language models locally on edge hardware.
• Grammar-Constrained Samplers - Enforces specific JSON schemas or grammar rules on model output to ensure valid response formats.
• On-Device Inference Executions - Runs machine learning model predictions directly on local hardware accelerators across various operating systems.
• Tool-Use Integrations - Allows defining external functions via specifications that the model can invoke to perform actions or fetch data.
• Constrained Decoding for Tool Use - Implements constrained decoding to facilitate precise function calling and agentic workflows.
• Schema-Constrained Sampling - Implements token selection restrictions based on JSON schemas or grammar rules during model inference.
• Persona and Behavioral Instructions - Allows the definition of system instructions and personas to control model behavior during conversations.
• Browser-Based Model Inference - Executes language models directly in the web browser using WebGPU for hardware-accelerated local text generation.
• Conversation State Management - Maintains conversation state and dialogue history to provide context across multiple turns in a session.
• State-Tracking Dialogue Managers - Tracks interaction history and applies system instructions to maintain dialogue continuity across multiple messages.
• Tool Integrations - Maps model-generated tool requests to external client-side functions for real-time data retrieval.
• External Tool Execution - Defines custom logic and parameters that the model can automatically invoke to perform tasks or fetch data.
• Function Calling Interfaces - Integrates external functions into on-device models to perform specific actions or retrieve real-time data.
• Automatic Tool Executions - Automatically invokes predefined external functions based on model requirements using docstrings and type hints.
• Apple Silicon GPU Accelerators - Integrates language models into Apple ecosystems with specific GPU acceleration for M-series hardware.
• In-Browser Model Execution - Executes language models directly in the browser using WebGPU for hardware-accelerated inference.
• Prompt Templates - Uses template engines to format raw input into structured prompts compatible with instruction-tuned models.
• Interactive Agent Chat Interfaces - Provides interfaces for real-time, conversational interaction with local AI models.
• Browser-based Inference Engines - Executes large language models directly in the web browser using WebGPU hardware acceleration.
• Apple Ecosystem Integration - Optimizes models for native deployment and GPU acceleration within Apple operating systems.
• Model Capability Extensions - Provides integrations for multi-token prediction and function calling to enhance the functional output of local models.
• Conversational Context Initialization - Defines the starting state of conversations using system instructions and initial examples.
• Grammar-Constrained Generation - Enforces JSON schemas or grammar rules on model responses to ensure predictable and valid output formats.
• Model Response Streaming - Streams model outputs incrementally to the client for a more responsive user experience.
• OpenAI-Compatible Model Servers - Hosts a local HTTP server that implements the OpenAI API specification for seamless model integration.
• On-Device Tool Integration - Connects local language models to external functions and APIs for real-time data retrieval and action execution.
• Prompt Templates - Uses a template engine to format raw inputs into structured prompts compatible with specific models.
• Constrained Decoding - Provides constrained decoding to ensure model outputs follow specific structured formats via logit manipulation.
• Structured Output Enforcements - Restricts model responses to specific formats using JSON schemas, regular expressions, or grammar rules.
• Speculative Decoding - Increases token generation speed on GPU backends by predicting multiple tokens simultaneously via speculative decoding.
• Tool-Calling Mappings - Maps model-generated requests to external functions using descriptive docstrings and type hints for real-time data retrieval.
• Token Mixing Accelerators - Reduces inference latency on mobile GPUs by employing multi-token prediction strategies.
• Inference Engine API Bridges - Provides native API interfaces for integrating on-device model inference into cross-platform applications.
• LLM Inference Plugins - Provides a dedicated plugin to run large language models locally in Flutter-based mobile and desktop applications.
• On-Device LLM Flutter Plugins - Provides a dedicated Flutter plugin for executing large language models locally on mobile and desktop.
• API Emulators - Hosts a local server that mimics standard API endpoints to integrate on-device inference into existing software workflows.
• OpenAI-Compatible Servers - Implements a local server mimicking OpenAI API endpoints to ensure compatibility with existing AI software workflows.
• OpenAI-Compatible API Servers - Provides a local HTTP server that implements the OpenAI API specification for drop-in model serving.
• Inference Engines - Production-ready framework for deploying LLMs on edge devices.
• Model Serving & Deployment - Deploys LLMs specifically on edge devices.

Star history