microsoft/onnxruntime

Features

• Cross-Platform Inference Frameworks - Enables consistent machine learning model execution across diverse operating systems and hardware architectures.
• Hardware Acceleration - Offloads model computation to specialized hardware like GPUs and NPUs to improve inference speed.
• Edge AI Model Deployment - Optimizes and deploys machine learning models for efficient execution on local edge and mobile hardware.
• ONNX Runtime Inference - Executes pre-trained machine learning models using a standardized, cross-platform runtime for hardware-accelerated performance.
• Graph-Based Execution Engines - Optimizes and executes machine learning models by representing them as directed acyclic graphs of tensors and operators.
• Cross-Platform Runtimes - Provides a cross-platform runtime environment for executing machine learning models consistently across diverse hardware and operating systems.
• Generative AI Integrations - Integrates generative AI capabilities including tokenization, chat templates, and streaming output into applications.
• Generative AI Models - Provides a framework for managing tokenization, chat templates, and streaming generation loops for large language models.
• Machine Learning Engines - Enables building minimized, optimized inference engines by selecting specific operators and hardware providers.
• Model Exporters - Converts trained machine learning models into standardized formats for high-performance cross-platform execution.
• ONNX Model Exporters - Converts pre-trained machine learning models into the standardized ONNX format for cross-platform execution.
• Natural Language Processing - Converts raw text into numerical sequences compatible with language models using standard vocabulary mapping and subword splitting.
• Provider Abstractions - Normalizes diverse hardware-specific API schemas into a unified format to decouple model logic from hardware kernels.
• Sequence Learning Models - Produces output sequences from a model based on provided generator parameters and the current model state.
• Language Model Integrations - Provides language-specific bindings to execute pre-trained models within native applications.
• Chat Template Management - Organizes input data into structured chat formats and enforces output schemas for conversational interaction and tool calling.
• Inference Execution Models - Executes pre-trained machine learning models on mobile devices for real-time tasks like object detection.
• Model Inference Servers - Hosts machine learning models in high-performance server environments to manage concurrent requests and optimize production throughput.
• Tensor Operations - Creates multi-dimensional arrays with specific shapes and data types to serve as inputs or outputs for model operations.
• Model Adapters - Switches between different fine-tuned model adapters at runtime to modify behavior dynamically without changing the base model.
• Model Capability Extensions - Integrates custom mathematical operations and specialized computational layers into the inference engine to support non-standard model architectures.
• Model Execution Environments - Runs generative inference loops including tokenization, model execution, and cache management.
• Model Graph Optimizers - Simplifies and optimizes computational graphs by fusing operators and pruning redundant nodes to improve inference performance.
• Model Optimization Toolkits - Offers a toolkit for model quantization, precision conversion, and graph-level transformations to improve inference performance.
• Text Tokenizers - Encodes input text into numerical sequences and decodes generated sequences back into human-readable text using specialized tokenization logic.
• Token Streamers - Delivers generated tokens incrementally to the user interface as they are produced to provide real-time feedback.
• Inference Engine Compilers - Supports building custom, hardware-specific inference engines from source to optimize performance for target environments.
• GPU Acceleration - Offloads model computation to NVIDIA graphics hardware to increase throughput and reduce latency.
• Operation Execution - Processes models using a wide range of standardized operator sets and supports custom operator execution.
• Data Preprocessing - Transforms raw inputs like text or images into tensor formats required by models using integrated operators.
• Model Deployment Pipelines - Streamlines the development, optimization, and deployment of machine learning models using automated workflows.
• Web-Based Model Deployment - Generates web-based artifacts for browser-based model execution with hardware acceleration support.
• Machine Learning Model APIs - Executes models from common frameworks by converting them to a standardized format for cross-platform inference.
• Hardware Acceleration - Configures performance modes, precision, and memory strategies to balance inference latency and accuracy.
• Model Loading - Initializes models from file paths to prepare them for inference and sequence generation.
• Model Execution Interfaces - Provides a consistent programming API across languages and hardware to facilitate switching between acceleration backends.
• On-Device Models - Executes machine learning models on mobile devices using native interfaces for local inference.
• Token Decoders - Converts numerical token sequences or streams of tokens back into human-readable text strings for display.
• Machine Learning - Training and inference for ONNX models.
• Custom Operator Interfaces - Provides mechanisms for registering and integrating user-defined mathematical operations into the core computation pipeline.
• Cloud Deployment - Packages machine learning models and scoring logic into scalable web services to provide real-time predictions via network requests.
• Cloud Infrastructure Deployment - Executes machine learning models within managed cloud environments and containerized services to scale inference across distributed systems.
• Model Conversion - Transforms machine learning models into optimized formats for high-performance execution across diverse hardware.
• Hardware Acceleration - Integrates custom hardware acceleration by implementing standardized interfaces that delegate sub-graph execution to specific devices.
• Chipset Acceleration - Executes models on mobile or embedded chipsets by offloading computation to dedicated processor cores.
• Language Bindings - Generates native language interfaces to enable model execution within diverse programming environments.
• Deployment-Specific Runtimes - Enables the creation of minimized, deployment-specific inference runtimes tailored to constrained environments or specific models.
• Mixed Precision Training - Balances model accuracy and performance by automatically identifying and retaining critical operations in higher precision.
• Model Performance Benchmarking - Measures and compares the accuracy and latency of model inference across different execution environments and hardware backends to validate performance gains.
• Model Architecture - Configures metadata, tokenization, and structural submodules to ensure compatibility with specific generative and multi-modal model architectures.
• Model Fine-Tuning - Supports on-device model updates and fine-tuning through training-specific APIs and library compilation.
• Model Fine-Tuning and Adaptation - Applies pre-compiled adapter binaries to model graphs for efficient task-specific adaptation during inference.
• Sequence Encoders - Transforms strings or batches of strings into numerical sequences compatible with model processing requirements.
• Model Conversion Utilities - Reduces model size and improves inference speed by casting floating-point operations to half-precision.
• Model Quantization - Matches weights and activation tensors between original and quantized models to resolve precision loss.
• On-Device Inference Engines - Supports on-device training and local adaptation of machine learning models on edge hardware.
• Tensor Data Accessors - Provides direct access to tensor memory, shapes, and data types for custom input and output manipulation.
• Operator Registries - Adds user-defined operations to the engine by linking shared libraries or calling registration functions.
• Cross-Platform Build Targets - Supports cross-compilation and native builds for specialized hardware platforms, including non-x86 architectures.
• Deployment Targets - Compiles the library for mobile and web browser environments to ensure compatibility with diverse deployment targets.
• Runtime Configurations - Accesses backend-specific settings and hardware properties through structured configuration strings to tune performance for specific environments.
• Arena Allocators - Manages tensor memory through pre-allocated pools to eliminate costly runtime allocations during iterative inference loops.
• Python Bindings - Provides native integration layers for Python environments to facilitate model execution.
• Integer Conversion Utilities - Reduces model size and accelerates inference by mapping 32-bit floating point values to lower-precision integer representations.
• Executable Footprint Optimizers - Customizes the set of included operator kernels during the build process to reduce the final executable footprint.
• Multimodal Input Processors - Loads and processes image and audio data into tensors for use as inputs within generative model inference pipelines.
• Chat State Management - Maintains conversational state by iteratively appending user input to token sequences for continuous model interaction.
• Chat Generation Strategies - Applies search algorithms and sampling methods like TopP or TopK to manage output generation.
• Generation Controls - Adjusts search strategies and guidance settings to control model behavior and influence generated output.
• Hardware Acceleration Backends - Checks for the availability of specific hardware backends to determine if accelerated model execution is supported on the current device.
• Model Inference Runtimes - Encapsulates external models or inference engines within custom operators to integrate third-party formats.
• Performance Profilers - Analyzes model performance by identifying computational bottlenecks and measuring the time spent on individual operators or subgraphs during inference.
• Model Output Formatting - Processes raw model results into application-specific formats for use in user interfaces.
• Model Training Optimizers - Provides pre-configured environments with optimized libraries to accelerate model execution and deployment on cloud infrastructure.
• Bundle Size Optimization - Minimizes binary footprint by including only necessary operators for specific model requirements.
• Operator Packages - Bundles specialized machine learning operations into installable packages for Python, Java, and other environments.
• Library Packaging Tools - Generates custom-built library packages for mobile platforms to support optimized native application integration.
• Android Packages - Generates archive files from model definitions to enable direct integration and deployment within mobile development environments.
• Windows Deployment Utilities - Facilitates native machine learning model execution on Windows using platform-specific or cross-platform interfaces.
• Binary Swapping Utilities - Enables the replacement of default library implementations with custom builds to support specific hardware execution providers.
• Cross-Language Bindings Layers - Provides bridging mechanisms to map low-level C interfaces to multiple programming languages for consistent model execution.
• Context Binaries - Converts models into optimized context binaries tailored for specific hardware architectures.

Star history