pytorch/executorch

Features

• Backend-Specific Model Exports - Delegates model graphs to hardware-specific backends like Core ML and QNN for accelerated execution.
• Cross-Language ML Toolkits - A toolkit that packages and deploys machine learning models to Android, iOS, and bare-metal RTOS environments with multi-language bindings.
• On-Device Decoder-Only Runners - ExecuTorch loads a decoder-only model and generates text from a prompt, streaming tokens as they are produced.
• On-Device Text Generation Runners - ExecuTorch loads a text-generation model, configures its tokenizer, and generates token streams on iOS through a native Objective-C/Swift interface.
• Generation Parameter Configurations - ExecuTorch controls generation behavior by setting parameters such as token count, temperature, and echo mode.
• NPU Inference Execution - Executes inference on CPU, GPU, NPU, or DSP by selecting a backend matching the target hardware.
• Hardware Acceleration Backends - Ships a hardware acceleration backend system that offloads model execution to NPUs and other accelerators.
• Hardware Backend Targeting - Selects and configures target hardware backends to lower model operations for efficient on-device execution.
• Hardware-Specific Model Exports - Exports and optimizes models for specific hardware backends, producing specialized files for efficient device execution.
• On-Device Inference - Executes machine learning models directly on mobile, embedded, and edge hardware for real-time, privacy-preserving applications.
• KV Cache Optimizations - Enables key-value caching and scaled dot-product attention to accelerate autoregressive generation.
• Model Execution APIs - Executes loaded models with input tensors and retrieves output tensors from inference.
• Edge AI Model Deployment - Exports, optimizes, and runs trained models on mobile, embedded, and edge hardware with a lightweight runtime.
• On-Device Deployments - Exports, optimizes, and runs large language models on resource-constrained edge devices for on-device text generation.
• On-Device LLM Runners - ExecuTorch exports and runs large language models on edge devices with a dedicated runner for text generation.
• iOS Deployments - ExecuTorch loads and executes a compiled model from Objective-C, integrating inference into iOS applications.
• Inference Execution Models - Executes forward passes or named methods on loaded models, returning output tensors or error codes.
• Hardware-Specific Model Optimizations - Optimizes and lowers models for specific hardware backends, producing specialized files for efficient device execution.
• Edge Hardware Optimizations - Applies graph optimizations, quantization, and caching to reduce inference latency and memory footprint for edge devices.
• Model Loading - Loads serialized models from file paths with a single constructor call for execution.
• On-Device Model Loaders - ExecuTorch loads a pre-trained LLM into memory on an Android device using a Java interface, preparing it for inference.
• Edge Model Compilers - Compiles and quantizes PyTorch models into compact binaries tailored for resource-constrained edge hardware.
• Compilation Pipelines - Provides an ahead-of-time compilation pipeline that converts PyTorch models into executable programs for edge hardware.
• Model Graph Optimizers - Applies operator fusion, decomposition, and backend-specific lowering to improve inference performance.
• Model Quantization - Applies quantization and optimization to reduce model size and improve inference speed on constrained hardware.
• Model Quantization - Applies quantization strategies during export to reduce model size and speed inference.
• Canonical Operator Sets - Defines a canonical operator set and graph representation that decouples model authoring from target-hardware execution.
• Compact Binary Serializations - Serializes trained models into compact binary files that the runtime loads and executes on target hardware.
• Large Language Model Exports - Exports large language models to optimized programs using a high-level API with configuration.
• On-Device Inference Exports - Exports large language models to portable files with built-in optimizations for mobile and edge hardware.
• Text Tokenization - ExecuTorch sets the tokenizer that converts text to model tokens and back, enabling the model to process and generate human-readable text.
• On-Device Inference Engines - Runs large language models directly on mobile and embedded hardware for text generation without cloud connectivity.
• On-Device Models - Loads and runs compiled programs on mobile or embedded hardware for efficient inference.
• LLM Runtimes - A runtime environment for executing large language models on-device with tokenization, KV cache management, and streaming text generation.
• Local LLM Execution - Loads and executes exported LLMs on Android and iOS through C++ APIs or platform-specific bindings.
• Operator Standardization - Specifies a canonical collection of ATen operators that the runtime supports for consistent behavior across deployments.
• PyTorch Model Export - Captures trained model computation graphs using standard PyTorch export APIs for deployment outside training.
• Dialect-Based Exports - Converts PyTorch models through ATen and Edge dialects into serialized binaries for on-device execution.
• Edge Device Exports - Converts PyTorch programs into portable graph representations using torch.export for edge device deployment.
• Edge Format Exports - Converts PyTorch models into portable .pte files for deployment on mobile and embedded hardware.
• On-Device Inference Exports - Converts PyTorch eager-mode models into portable intermediate representations for edge hardware execution.
• Static Graph Exports - Captures PyTorch programs as static graphs of standardized operators for deployment on resource-constrained devices.
• Weight Quantization - Converts model weights to lower-precision formats like int8 or int4, configurable per backend.
• On-Device Text Generation Runners - Manages tokenization, KV cache, and streaming for on-device large language model inference sessions.
• Static Graph Compilers - Exports, quantizes, and lowers model graphs into executable programs tailored for specific edge hardware targets.
• Subgraph Backend Delegation - Provides a standardized interface for third-party compilers to compile and execute subgraphs on accelerators.
• LLM Quantization Exports - Applies TorchAO or pt2e quantization to reduce model size and improve execution speed on resource-constrained hardware.
• Large Language Model Deployments - Runs large language models on-device for text generation without requiring a cloud connection.
• On-Device LLM Runners - ExecuTorch loads and executes a text-generation LLM on-device, handling tokenization and autoregressive generation.
• Custom Backend Integrations - ExecuTorch integrates a new hardware or software backend into the compilation and runtime pipeline by providing partition, preprocess, and execution functions.
• Model-as-a-Table Integrations - ExecuTorch links only the kernels and backend libraries required by a specific model to produce a small, efficient executable for deployment.
• On-Device LLM Session Runners - ExecuTorch loads and executes a language model on-device, wrapping the runtime for text generation tasks.
• Model Executions - ExecuTorch loads a serialized program and runs it through a lightweight C++ runtime that dispatches operations to the appropriate kernels or backends.
• Model Deployment Platforms - Integrates exported models into Android, iOS, and desktop applications using platform-specific runtime bindings.
• AI Model Execution - Loads serialized programs and runs inference with a lightweight C++ runtime on edge devices.
• Mobile Model Deployment - Runs exported models on Android or iOS using platform-specific APIs and sample applications.
• LLM Mobile Deployments - ExecuTorch executes a deployed large language model on Android or iOS hardware, including optional acceleration via Qualcomm AI Engine Direct.
• Mobile Framework Integrations - ExecuTorch bridges the runtime with mobile toolkits so AI models run natively inside mobile applications.
• Token Streaming - ExecuTorch produces output tokens one at a time from a loaded model, streaming the generated text as it is produced.
• Fixed-Size Memory Pools - ExecuTorch provides a fixed-size memory pool that the runtime uses to allocate intermediate tensors during model inference.
• Operator Kernel Selective Linking - ExecuTorch links only the kernels and backend libraries required by the deployed model to minimize the application's binary footprint.
• Static Memory Planning - Plans memory allocation statically during compilation to avoid dynamic allocation overhead at runtime on edge devices.
• Model-Specific Memory Planning - Pre-allocates memory buffers for tensors at compile time to avoid runtime allocation overhead and reduce peak memory usage.
• Inference Runtime Binary Reduction - ExecuTorch links only the kernels and operators required by a specific model, reducing the binary size for deployment on resource-constrained devices.
• C++ Inference Runtimes - ExecuTorch links the inference engine into a C++ application and manages model loading, tensor memory, and custom module extensions.
• C++ Model Interfaces - ExecuTorch loads and runs a compiled model using C++ APIs, enabling inference on embedded systems with or without dynamic memory.
• Model Program Loading - Loads serialized model programs from files or data sources for execution on target devices.
• Model Method Initializations - Runs named methods on loaded models, passing tensors and retrieving outputs from the runtime.
• Backend-Enabled Builds - ExecuTorch activates support for specific hardware accelerators by setting build flags for each backend.
• Intermediate Representations - Defines a portable graph format that decouples model authoring from target-hardware execution.
• Multi-Platform Mobile Deployers - Integrates and runs machine learning models on Android, iOS, and microcontrollers with multi-language runtime bindings.
• Computation Subgraph Delegation - A system that partitions model subgraphs and offloads computation to accelerators like NPUs, GPUs, and DSPs.
• Platform Abstraction Layers - ExecuTorch isolates platform-specific system calls behind a thin abstraction layer so the runtime can be re-targeted without changing model code.
• Apple Hardware Acceleration - Accelerates model inference on iOS devices using Apple-specific hardware backends for execution.
• XNNPACK Backend Exports - Converts PyTorch models into programs lowered for the XNNPACK delegate during export.
• Computational Graph Visualizations - Displays the model as a graph with performance and debug data linked back to original source code.
• Model Deployment - Executes image classification, object detection, and other vision tasks directly on edge hardware.
• Computer Vision Preprocessing - ExecuTorch resizes, crops, converts, and normalizes image tensors for model inference on Android and iOS.
• Dynamic Tensor Shapes - Handles tensor dimension changes between inference runs without requiring model recompilation.
• GPU-Accelerated Inference - Executes machine learning models on compatible GPUs using the cross-platform Vulkan API for accelerated inference.
• Half-Precision Inference - Supports FP16 inference precision to balance model accuracy and memory usage on edge devices.
• Hardware-Accelerated Inference - Delegates model execution to Qualcomm Hexagon or Adreno processors through the QNN SDK for on-device AI.
• Backend Dialect Translation - Translates portable model graphs into hardware-specific representations for execution on accelerators.
• Image Data Preprocessing - ExecuTorch keeps platform-dependent image work like decoding, resizing, and cropping in the application layer before passing pixels to the exported model.
• Intel Hardware Export - Exports and executes models on Intel CPUs, GPUs, and NPUs using the OpenVINO runtime backend.
• Embedded Deployments - Runs ML models on microcontrollers and constrained hardware with DSP and NPU acceleration.
• Memory-Mapped Loading - Loads program files using memory mapping with optional locking and performance hints.
• Hugging Face Converters - Downloads transformer models from Hugging Face Hub and converts them into portable formats for on-device execution.
• Performance Profilers - ExecuTorch measures runtime performance of individual operators and layers to identify bottlenecks and guide optimization efforts.
• Disk Size Reducers - Shrinks model file size through quantization-aware and post-training quantization for edge deployment.
• Input-Output Tensor Assignments - Assigns input tensors to methods before execution and optionally pre-allocates output tensors.
• CPU Inference Quantizers - Runs models with 8-bit, fp32, or fp16 activations to reduce memory and improve speed on CPU hardware.
• Quantized Linear Layers - Executes linear layers with 8-bit or 4-bit weights to reduce memory and compute during inference.
• Hardware-Specific Quantizations - Applies static 8-bit or 16-bit integer quantization to models so they can run on Samsung Exynos hardware.
• Configurable Bit-Width Quantizers - Configures model quantization to specific bit-width combinations like A16W16 or A8W4 before deployment.
• Quantization-Aware Training - Supports quantization-aware and post-training quantization to shrink models for constrained hardware.
• Multimodal Model Runners - ExecuTorch loads and executes a multimodal model that accepts text, image, and audio inputs on iOS through a native Objective-C/Swift interface.
• On-Device Multimodal Runners - Ships a runtime that executes multimodal models on CPU, GPU, and Metal backends for on-device AI tasks.
• NPU Accelerators - Exports and lowers PyTorch models to run on MediaTek Neuron Processing Units for on-device inference.
• Samsung Exynos NPU/DSP Delegates - ExecuTorch delegates model execution to Samsung's on-device NPU or DSP via the Exynos AI Litecore SDK for hardware-accelerated inference.
• On-Device Model Profilers - ExecuTorch measures model load time, operator-level execution, delegate execution, and end-to-end inference latency during on-device execution.
• Precision Quantization - Converts tensors to lower-precision types using static, dynamic, or hybrid quantization techniques.
• Hardware-Specific Quantizations - Applies post-training or quantization-aware training to reduce model precision for efficient execution on Qualcomm accelerators.
• Debugging And Profiling - Captures runtime execution traces and links them back to original source code for performance analysis and numerical validation.
• LLM Profiling Suites - Generates operator delegation tables, memory profiles, and time profiles to diagnose exported LLM performance.
• Model Execution Debuggers - ExecuTorch inspects performance data, visualizes the graph, and correlates runtime behavior back to the original source code during development.
• On-Device Model Debuggers - ExecuTorch uses developer tools to inspect, visualize, and diagnose the behavior of a model during on-device execution.
• Custom Inference Operations - Accelerates inference by replacing standard attention and KV cache operations with faster custom implementations.
• Runtime Profiling Data Extractors - ExecuTorch captures profiling and debugging data from model execution and exposes it through a structured dump for post-run analysis.
• Arbitrary Source Model Loaders - Provides an abstraction to load model files from files, memory, or other data sources.
• Profiling Data Inspector APIs - ExecuTorch generates profiling artifacts and uses an inspector API to analyze operator-level performance and identify bottlenecks.
• Model Metadata Retrieval - Retrieves method names and tensor metadata like shape, type, and size from loaded models.
• LLM Component Customizations - ExecuTorch replaces default tokenizer, sampler, or acceleration backend with a user-defined implementation to adapt the inference pipeline.
• Intermediate Output Inspection - ExecuTorch inspects intermediate outputs and detects numerical discrepancies between ahead-of-time and runtime execution to validate model correctness.
• Operator-to-Source Tracebacks - ExecuTorch links each runtime-executed operator back to the exact line of Python code that produced it, enabling hotspot identification.
• Event-to-Source Mapping - ExecuTorch associates captured runtime events with model Python source code using an optional export-time record.
• Bundled Model Output Validators - ExecuTorch bundles a model with sample inputs and expected outputs to automatically verify that runtime results match expectations.
• LLM Module Builders - ExecuTorch constructs a language model inference module by specifying initialization parameters through a builder pattern.
• Android Hardware Deployments - Pushes compiled models and runtime libraries to Android devices for inference via the QNN backend.
• Model Executions - Runs exported ML models on Android devices with hardware acceleration for on-device inference.
• Single-Command Model Exports - Exports supported large language models to optimized formats using a unified CLI or YAML configuration.
• Model Inference Run Executions - ExecuTorch runs a pre-built program that loads a model file with random inputs to verify execution on a device.
• Export Pipeline Profiling - ExecuTorch generates detailed logs of operator delegation, memory usage, and export timing to diagnose performance bottlenecks.
• OpenVINO Model Exports - Exports PyTorch models to an optimized format using the OpenVINO toolkit for Intel hardware.
• Inference Runtimes - Provides a runtime that executes models on Arm Ethos-U NPUs with optimized kernels.
• Token Generation Callbacks - ExecuTorch receives each generated token via a callback function, enabling real-time output display during inference.
• Operator Kernel Implementations - ExecuTorch adds or replaces operator implementations in the kernel library to support specialized hardware or logic.
• Model Memory Configurations - ExecuTorch configures memory allocation, placement, and data loading directly for advanced hardware-specific use cases.
• Backend Binary Compilation - Transforms tagged model subgraphs into compiled binary blobs for direct loading onto target hardware.
• Multi-Language Runtime Bindings - Provides inference APIs for Java, Kotlin, Objective-C, and C++ to integrate the engine into diverse application environments.
• Model Graph Passes - Ships a custom pass system for applying user-defined transformations to model graphs during compilation.
• Runtime Module Integrations - ExecuTorch adds custom operators or modules to the runtime to support model-specific or hardware-specific logic.
• Runtime Feature Extensions - ExecuTorch includes extra runtime features like data loading, tensor management, or LLM support via build options.
• Runtime Customizations - ExecuTorch adapts the on-device runtime through a platform abstraction layer and custom integration code.
• Execution Tracing - Generates trace files during program execution to record debug and performance data for analysis.
• Post-Export Debug Linkers - ExecuTorch links runtime performance and debug data back to the original Python source code for post-execution analysis.
• Delegated Execution Mappers - ExecuTorch maps runtime errors or profiling data from a delegated subgraph back to the original PyTorch code.
• Compiled Model Validations - ExecuTorch loads a compiled file with Python bindings and runs inference to verify model accuracy before deployment.
• On-Device Runtime Debugging - Captures runtime performance, memory usage, and numerical accuracy of models executing on target hardware.
• On-Device Model Profilers - ExecuTorch instruments model execution to collect performance metrics and diagnose issues during inference on the target platform.
• On-Device Sanity Checks - ExecuTorch runs a pre-built executable that loads a model file and executes it with random inputs for quick validation.
• Multimodal Input Processors - ExecuTorch accepts image and audio data alongside a text prompt for models that support multiple input modalities, then generates a response.
• Inference Engines - On-device AI deployment framework for mobile and embedded systems.

Star history