openvinotoolkit/openvino

Features

• Inference Execution Engines - Acts as a high-performance runtime for executing optimized deep learning models across diverse hardware via a unified API.
• Model Serving - Hosts classic, generative, or graph models via REST or gRPC endpoints for remote inference.
• Generative - Provides a specialized execution environment for large language models featuring continuous batching and speculative decoding.
• CPU Optimizations - Tunes CPU resource usage through latency hints, thread count control, and CPU pinning for optimized inference.
• Cross-Hardware Model Inference - Executes deep learning models across heterogeneous hardware including CPUs, GPUs, and NPUs.
• Distributed Model Execution - Splits a single model across multiple accelerators to process operations on GPUs or NPUs.
• NPU Inference Execution - Provides specialized execution support for running deep learning inference on Neural Processing Units (NPUs).
• Hardware Acceleration - Manages the distribution of inference workloads across different hardware targets to maximize performance.
• Hardware Device Management - Coordinates model execution across different hardware devices using automatic selection and batch modes.
• Inference Device Discovery - Lists all compatible computing devices and identifiers to enable targeted model deployment across diverse hardware.
• Hardware Abstraction Layers - Provides a plugin-based abstraction layer that maps neural network operations to various hardware vendor drivers.
• Hardware Acceleration - Leverages specialized hardware accelerators like NPUs and GPUs to optimize model execution speed.
• Model Compilation - Transforms trained models into optimized versions specifically prepared for efficient inference execution.
• Model Optimization Toolkits - Provides a comprehensive toolkit for converting, quantizing, and compressing models from PyTorch, TensorFlow, and ONNX.
• Model Pipelines - Executes sequences of models and transformation nodes as a single unit for complex workflows.
• Model Quantization - Converts deep learning models to 8-bit precision using calibration datasets to significantly reduce model size.
• Model Serving Servers - OpenVINO hosts AI models for inference using container images or binaries to serve requests across environments.
• Weight Quantization - Employs microscaling quantization to reduce the memory footprint of large language models while maintaining high accuracy.
• Post-Training Quantization - Reduces model size and resource consumption by converting weights to 8-bit precision without needing to retrain.
• Stateful Model Execution - OpenVINO maintains internal state across consecutive inference requests to support models with temporal dependencies.
• Text Completion Engines - Produces text responses from prompts using unary or streaming calls for chat-based interfaces.
• Text Tokenizers - Converts raw strings into token IDs using model-specific tokenizers with padding and truncation.
• Inference Runtime Integrations - Serves as a high-performance runtime backend for other tools to execute optimized models with hardware acceleration.
• ML Model Hosting - Hosts machine learning models on dedicated servers or clusters to offload heavy computation from clients.
• Model Conversion - Transforms models from various frameworks into an optimized intermediate representation for efficient hardware execution.
• Generative - Provides specialized conversion of generative models into optimized intermediate representations for efficient execution.
• Model Serving Platforms - Provides a high-performance server to host and expose deep learning and generative AI models via REST and gRPC.
• Server-Side Tokenization - Handles tokenization and detokenization on the server side, allowing clients to send and receive raw strings.
• Device Selection - Automatically selects the optimal hardware device for a model with built-in fallback mechanisms.
• Inference Acceleration Drivers - Implements a plugin-based system that maps neural network operations to vendor-specific hardware drivers.
• Intermediate Representations - Transforms models from various frameworks into a standardized internal representation for hardware-agnostic optimization.
• Directed Acyclic Graph Engines - Orchestrates sequences of models and custom logic nodes as a directed acyclic graph for complex workflows.
• Model Pipeline Orchestration - Orchestrates a sequence of models as a directed acyclic graph to process a single request.
• OpenAI-Compatible APIs - Exposes chat and completion endpoints that follow the OpenAI API specification for ecosystem compatibility.
• Audio Transcription - Executes Whisper models on NPUs to convert spoken audio into text transcripts.
• Batch Size Tuning - Sets the number of input samples processed in a single pass based on fixed or automatic values.
• Automatic Batch Size Optimization - Automatically applies optimal batch sizes based on hardware to maximize total inference throughput.
• Document Rerankers - Provides utilities to sort document lists based on relevance to a query to improve retrieval accuracy.
• Dynamic Tensor Shapes - Allows modifying batch size and input shapes during execution to optimize the balance between throughput and latency.
• Embedding Generators - Provides endpoints for generating vector representations of text to support retrieval workflows.
• Chaining Pipelines - Passes output tensors from one model directly as input to another to create sequential pipelines.
• Image Generation - Deploys models that generate visual content from text prompts through a standardized API.
• Image Editing - Executes image-to-image and inpainting tasks via REST endpoints to modify existing visual content.
• Keras Backend Accelerators - Integrates a high-performance backend into the Keras workflow to speed up model execution on compatible hardware.
• Asynchronous - Triggers model execution in a non-blocking manner to process other tasks during computation.
• Inter-Model Data Transformation - Executes custom logic via dynamic libraries to convert data when one model output is incompatible with another input.
• RAG Pipelines - Acts as the inference engine for retrieval-augmented generation workflows via a compatible API.
• Model Construction - Constructs models by assembling pre-compiled operations to define inputs, outputs, and the computational graph.
• Stateful - Allows creating models from scratch by defining variables and operations to manage internal memory buffers.
• Mixed-Precision Computing - Supports floating-point and quantized data types for internal primitives to balance performance and accuracy.
• Quantized Model Deployments - Converts and compiles quantized models into intermediate representations for efficient execution on target hardware.
• Model Performance Benchmarking - Evaluates deployed model performance using industry-standard benchmarks for chat and completion tasks.
• ONNX Engine Conversions - Transforms models from the ONNX format into an optimized intermediate representation for efficient execution.
• Continuous Batching Strategies - Implements continuous batching to dynamically group asynchronous inference requests and maximize hardware utilization.
• Speculative Decoding Strategies - Accelerates token generation using a lightweight draft model to propose candidates for validation by a larger model.
• Local Language Model Execution - Executes language models in the GGUF format directly from binary files on local compute resources.
• Incremental Inference Streaming - Processes data streams through a graph to generate continuous model outputs in real time.
• Tensor Memory Management - Implements high-performance data transfer using remote tensors and buffers for inference input and output.
• Pre-compiled Shape Management - Optimizes inference for common input dimensions by maintaining several pre-compiled model versions.
• Prompt Lookup Decoding - Accelerates token generation by identifying n-gram matches within the prompt.
• Adapter Fusion - Merges Low-rank Adaptation weights into the baseline model to eliminate extra computation during deployment.
• Quantization Error Mitigation - Fine-tunes models during the quantization process to mitigate precision loss when converting to 8-bit integers.
• Vision-Language Models - Deploys multimodal models capable of analyzing combined text and image inputs for visual reasoning.
• Model Compilation Optimizers - Adjusts NPU compiler optimization levels and performance hints to balance compilation speed and execution efficiency.
• Graph Compilation Caching - Stores compiled model blobs on disk to eliminate expensive runtime optimization during application startup.
• Model Lifecycle Management - Controls active models in the server by pulling assets or removing them to stop service.
• Model Pruning - Removes redundant model parameters through sparsity-aware training to minimize the computational footprint.
• Multi-GPU Distribution - Splits models across multiple GPUs to enable the execution of models that exceed the memory of a single card.
• Model Performance Analysis - OpenVINO collects performance measurement counters and graph information for each model layer to CSV and XML files.
• Precision Preservation - Converts models to 8-bit precision while preserving high-impact operations to maintain accuracy thresholds.
• Model Serialization - Saves converted models to files to reduce load latency and shrink storage size.
• Model Serializers - Exports optimized model weights to persistent storage to avoid repeating the compression process upon reloading.
• Model Server Clients - Provides a gRPC client to send data to a running model server and receive inference results.
• Dynamic Model Reloading - OpenVINO monitors the file system for model changes and automatically refreshes the serving list.
• Embedded Inference Libraries - Provides the ability to link model serving capabilities directly into applications as a shared library for low-latency in-process inference.
• Model Versioning - OpenVINO organizes multiple versions of a model in a directory structure to serve specific versions on request.
• Model Versioning Systems - Implements systems to control and switch between specific model versions to optimize resource consumption and performance.
• LoRA Adapter Loaders - Downloads and applies low-rank adaptation weights from remote repositories to supplement base models.
• Parallel Inference Orchestrators - Uses device-side streams to process multiple inference requests asynchronously and increase hardware utilization.
• Preprocessing Pipelines - Translates image preprocessing pipelines into model operators and embeds them directly into the model.
• Sampling Controls - Implements parameters like temperature and top-p to modulate output diversity during token generation.
• Prefix Caching - Stores and reuses key-value tensors for static prompt prefixes to reduce latency in generative AI tasks.
• Kernel Acceleration - Compiles PyTorch code into optimized kernels to accelerate model execution on compatible hardware targets.
• Sparsity-Aware Weight Compression - Compresses weights with high zero-element ratios to reduce memory bandwidth usage and accelerate matrix multiplication.
• Speech to Text Transcription - Hosts transcription models that transform audio input into written text across diverse hardware.
• Structured Output Enforcements - Constrains language model responses to specific formats like JSON schemas or regular expressions.
• Tensor Management Utilities - Retrieves and assigns data to model tensors using names or indices to prepare for inference.
• Tensor Reshaping - Rearranges the dimensions of a tensor to match the layout expected by the target model.
• Text Embedding Generators - Processes text to create normalized vector embeddings and rerank results with configurable pooling strategies.
• Text Tokenization Utilities - Converts raw text strings into numerical tokens and prepares the necessary input tensors for model consumption.
• Token Detokenization - Translates numerical tokens produced by generative models back into human-readable text strings.
• Tokenizer Intermediate Representations - Transforms Hugging Face tokenizers and detokenizers into an intermediate representation via CLI or API.
• Training-Time Compression - Executes compression algorithms alongside the training process to reduce model size and increase inference performance.
• Vision-Language Inference - Processes combined image and text inputs on NPUs to generate analytical text outputs.
• KV Cache Management - Reduces the memory footprint of attention tensors to support higher concurrency and longer sequences in generative AI.
• Hub-Integrated Deployment - Downloads, configures, and serves generative AI models directly from the Hugging Face hub without manual preparation.
• Zero-Copy Memory Mappings - Eliminates data duplication overhead by sharing memory buffers between the host and hardware accelerators.
• Image Preprocessing Utilities - Encodes binary image data to specific layouts and color formats while resizing inputs for model compatibility.
• Input Normalizers - Subtracts mean values and divides by standard deviations to normalize tensors for model inference.
• Inference State Management - Retrieves or resets internal memory values of a stateful model to control data dependencies.
• Zero-Copy Data Access - Uses shared memory and zero-copy tensors to avoid expensive data duplication during inference.
• Inference Batching - Groups multiple inference requests into single execution calls to maximize hardware accelerator utilization.
• Dynamic Configuration - Provides the ability to update model settings and device assignments at runtime without restarting the inference service.
• Variable Input Shape Support - Configures model input dimensions to be dynamic, allowing the handling of varying batch sizes or image resolutions.
• Remote Model Loading - Retrieves AI models directly from cloud storage using URI paths and authentication credentials.
• Containerized Model Serving - Deploys a model server within a container to serve models from cloud storage or local files.
• Inference Scaling Services - Expands inference capacity vertically through resource allocation or horizontally across multiple hosts.
• Kubernetes Orchestration - Orchestrates the deployment of inference services within Kubernetes clusters using hardware-acceleration configurations.
• Hugging Face - Transforms deep learning models from Hugging Face into a specialized intermediate representation for optimized execution.
• JAX and Flax - Transforms JAX and Flax model objects or traced functions into an intermediate representation for deployment.
• Keras - Exports models from any Keras backend into a standardized intermediate representation for optimized deployment.
• PyTorch - Transforms PyTorch model objects or files into an optimized intermediate representation via direct conversion.
• TensorFlow - Transforms models from various TensorFlow formats into an optimized representation for inference.
• TensorFlow Lite - Transforms .tflite model files into an optimized intermediate representation to reduce load latency.
• Quantization Parameter Tuning - Evaluates accuracy changes against a baseline to find the optimal quantization settings for a model.
• Distributed Device Orchestration - Splits single model execution across multiple computing devices to optimize hardware utilization.
• Ahead-Of-Time Compilation - Compiles model graphs into binary blobs on disk to eliminate expensive on-the-fly optimization during startup.
• Inference - Detects and replaces parameter-result pairs in models containing loops to reduce inference latency.
• Denial of Service Prevention - Protects system availability by limiting the number of parallel inference streams and workers to prevent resource exhaustion.
• Custom Graph Operations - Adds user-defined operations to the execution graph during the model compilation phase.
• Inference Stream Multiplexing - Processes inference requests simultaneously across multiple host threads while efficiently sharing model weight memory.
• Concurrency Tuning - Optimizes high-load performance by tuning worker counts and request queue sizes to prevent bottlenecks.
• Execution Time Profilers - OpenVINO analyzes execution hotspots and timing through instrumentation APIs to identify time-consuming functions.
• Color Space Converters - Transforms image color spaces and splits YUV formats into separate planes to match model requirements.
• Embedded Model Runtimes - Allows embedding model management and pipeline functionality directly into applications via a C API to eliminate network latency.
• Model Optimization and Acceleration - Toolkit for optimizing and deploying AI inference on Intel hardware.
• Model Serving & Deployment - Optimizes and deploys AI inference on Intel hardware.

Star history