google-ai-edge/LiteRT
LiteRT
LiteRT is a runtime and API for executing machine learning and generative AI models on mobile, desktop, and IoT hardware. It consists of an inference engine and a specialized environment for running quantized large language and diffusion models locally on edge hardware.
The system includes an ahead-of-time model compiler that translates models into hardware-specific bytecode to reduce startup latency and memory overhead. It provides a unified interface for Neural Processing Units with automatic fallback routing to CPUs or GPUs when specific subgraph support is unavailable. An edge model converter transforms trained models into optimized formats for deployment on resource-constrained devices.
The project covers model optimization through format conversion and post-training quantization to reduce binary size. It manages hardware acceleration through automatic accelerator selection and zero-copy memory optimizations to eliminate CPU memory copying. The framework also supports custom operator definitions through a low-level kernel interface to extend the supported mathematical operations.
Features
- Local Inference Engines - Ships a high-performance inference engine and API for executing machine learning and generative AI models on mobile, desktop, and IoT hardware.
- On-Device Inference Engines - Provides an inference engine optimized for executing machine learning and generative AI models locally on mobile, desktop, and IoT hardware.
- On-Device Deployments - Enables the deployment and execution of large language models and agentic planning capabilities entirely on-device.
- Generative AI Runtimes - Provides a specialized environment for running quantized large language and diffusion models locally on edge hardware.
- On-Device LLM Runners - Provides a specialized environment for running quantized large language and diffusion models locally on edge hardware.
- Edge Hardware Optimizations - Applies quantization and graph optimizations to reduce memory footprint and increase inference speed on resource-constrained hardware.
- Model Format Converters - Provides tools for transforming trained machine learning models into optimized formats for deployment on resource-constrained edge devices.
- Model Quantization - Applies quantization and architecture-specific optimizations to enable large language and diffusion models to run locally.
- NPU Acceleration - Integrates with specialized Neural Processing Units to optimize model inference performance and energy efficiency.
- NPU Accelerators - Implements a unified interface for executing machine learning inference on Neural Processing Units with automatic CPU and GPU fallbacks.
- Post-Training Quantization - Reduces model precision from floating point to integers after training to decrease binary size and increase inference speed.
- Model Bytecode Compilation - Translates high-level machine learning model definitions into optimized low-level bytecode for specific edge hardware architectures.
- Model Conversion - Transforms machine learning models into specialized formats to increase execution speed and reduce memory usage on edge hardware.
- Build-Time Bytecode Compilation - Translates models into bytecode during the build process to reduce runtime initialization and memory overhead on edge devices.
- Ahead-of-Time Wasm Execution - Compiles models into hardware-specific bytecode before deployment to minimize startup latency on constrained devices.
- Computation Subgraph Delegation - Partitions model computation graphs to delegate specific operations to the most compatible CPU, GPU, or NPU backends.
- Hardware-Accelerated Inference - Executes machine learning models on edge devices using CPU, GPU, or NPU hardware acceleration for high performance.
- Execution Fallbacks - Automatically redirects computation to a compatible processor if the primary hardware accelerator lacks support for specific operations.
- On-Device Inference - Provides a high-level inference API and runtime environment to manage model state on edge devices.
- Hardware Acceleration - Enables the execution of machine learning and generative AI models across mobile, desktop, and IoT hardware using CPUs, GPUs, and NPUs.
- Automatic Accelerator Selection - Automatically selects the most efficient hardware accelerator and manages asynchronous execution for machine learning tasks.
- Tensor Memory Management - Manages the allocation and tracking of memory views for tensors using buffer references to control data flow.
- Hardware Performance Tuning - Tunes execution across CPUs, GPUs, and NPUs using hardware-specific optimizations to achieve peak processing speeds.
- Graph Compilation Caching - Stores compiled computation graphs in a local directory to bypass runtime initialization overhead.
- NPU Unified Interfaces - Provides a unified interface to execute models on NPUs while abstracting vendor-specific compiler and runtime details.
- On-Device Compilation - Translates models into NPU instructions during application initialization to ensure compatibility across diverse hardware platforms.
- Hardware Buffer Zero-Copy - Eliminates expensive CPU memory copy operations by passing tensor data directly to the NPU hardware buffer.
- Model Artifact Caches - Caches pre-compiled model hardware instructions in local storage to eliminate repeated translation during application launches.
- Zero-Copy Buffer Interoperability - Passes tensor data directly to accelerators without duplicating data to system memory to reduce latency and power.
- PyTorch - Provides specialized paths for converting trained PyTorch models into optimized formats for on-device deployment.
- Inference Engines - Framework for efficient ML and GenAI deployment on edge.
- Model Serving & Deployment - Deploys models on mobile and edge devices.
