uxlfoundation/oneDNN

Features

• Deep Learning Acceleration - Provides optimized basic building blocks for neural networks to accelerate tensor operations across CPU and GPU hardware.
• High-Performance Scientific Computing - Runs optimized high-performance scientific computing primitives such as convolutions and LSTM operations using pre-computed state.
• Accelerated Normalization Layers - Provides high-performance implementations of batch, group, layer, and local response normalization to accelerate training.
• Activation Functions - Implements non-linear activation functions such as PReLU and Softmax to be applied element-wise to tensors.
• Backward Propagation Primitives - Calculates operation gradients using source or destination tensors to update weights during neural network training.
• Batch Normalization - Calculates forward and backward normalization on multi-dimensional tensors using mean and variance to stabilize training.
• Convolutional Operations - Executes high-performance forward, backward, and weight update passes for multi-dimensional convolutions, including grouped and dilated variants.
• Deep Network Training - Implements high-performance primitives for calculating gradients and performing backward propagation during deep network training.
• Just-In-Time Kernel Compilers - Uses just-in-time compilation to target specific processor instruction sets and maximize hardware-specific execution speed.
• Graph Fusions - Groups individual primitives into high-level graphs to automatically apply backend-aware operation fusions for better performance.
• Kernel Optimizers - Optimizes the dispatch of JIT-generated kernels by preferring specific CPU instruction set features for maximum performance.
• Convolution Layers - Implements convolutional and transposed convolutional layers for processing spatial data in forward and training modes.
• Tensor Memory Management - Manages tensor memory layouts, data shapes, and device allocations across different hardware engines.
• Hardware-Aware Graph Partitioning - Analyzes operations and decomposes them into optimized sub-graphs and partitions to maximize efficiency on specific hardware.
• Low Precision Inference - Executes deep learning operations using reduced-precision numerical types to increase throughput and reduce memory usage.
• Static Quantization - Maps quantized integers to floating-point values during primitive execution using pre-defined scales and zero-points.
• Dynamic Quantization - Computes scaling factors on the fly for groups of elements during inference to reduce precision and increase throughput.
• Primitive Accelerators - Provides tuned acceleration routines for basic neural network operations across CPU and GPU hardware.
• Inference Optimizations - Increases inference throughput and reduces memory bandwidth through operation fusion and low-precision numerical types.
• Quantized Neural Network Execution - Runs integer-based deep learning operations using scaling factors and zero points to reduce memory usage and increase speed.
• Recurrent Neural Networks - Provides high-performance primitives for processing sequential data using RNN, LSTM, GRU, and AUGRU architectures.
• Sparse Computing Kernels - Utilizes sparse memory formats and specialized kernels to reduce memory footprint and skip zero-value computations.
• Memory Layout Optimizations - Configures model memory formats and data layouts to maximize throughput on tensor cores and other accelerators.
• GPU Kernel Primitives - Implements highly tuned GPU kernels for fundamental deep learning operations like convolutions and matrix multiplications.
• Runtime Hardware Optimizers - Automatically detects host processor capabilities at runtime to select the most efficient instruction sets for acceleration.
• Tensor Layout Optimizations - Restructures multi-dimensional tensor layouts to align with physical memory and maximize hardware processing unit throughput.
• CPU Instruction Optimizations - Targets specific processor instruction sets like AVX-512 and AMX to maximize execution speed through low-level optimizations.
• CPU Primitive Execution - Runs deep learning operations on various processor architectures using just-in-time code generation for the detected instruction set.
• Device-Specific Memory Allocators - Provides specialized allocators for managing tensors in discrete device memory spaces and facilitating host-to-device transfers.
• ISA-Level Binary Generation - Lowers computation logic to hardware ISA level and generates specialized binary code based on tensor metadata.
• Runtime GPU Kernel Compilation Libraries - Compiles kernels online using just-in-time compilation to enable dynamic specialization for the detected hardware.
• Tensor Computation Graphs - Represents sequences of mathematical operations as directed graphs of tensors to optimize execution flow and dependencies.
• Matrix Multiplication Primitives - Executes matrix-matrix multiplication using floating-point, quantized, or sparse memory encodings across various data types.
• Kernel Implementation Selection - Selects between direct, Winograd, or implicit GEMM implementations to balance performance, memory, and numerical accuracy.
• Depthwise Separable Convolutions - Implements optimized depthwise separable convolutions by fusing depthwise and pointwise operations to accelerate separable convolution stacks.
• Dequantizing Runtimes - Implements on-the-fly conversion of integer weights back to floating-point format during computation.
• Dropout Regularization - Integrates dropout regularization directly into the primitive output buffer to prevent overfitting during neural network training.
• Mixture-of-Experts Inference Optimizers - Uses grouped memory formats and grouped GEMM to optimize variable-size batching for Mixture-of-Experts model inference.
• Primitive State Caching - Stores and reuses previously created execution primitives to avoid the overhead of repeated initialization.
• Operator Fusions - Reduces inference overhead by fusing neural network primitives with subsequent post-operations into single kernels.
• Optimized Partition Execution - oneDNN runs compiled binary code using bound data buffers and streams to process multi-dimensional tensors on a target engine.
• Quantization Granularity Settings - Provides configuration options to define the scope of quantization, ranging from global tensors to block-wise masks.
• Primitive Execution Configurations - Adjusts operational behavior through math modes and accumulation precision to balance performance and accuracy.
• Parameter Definitions - Defines specific quantization settings and parameters required for performing integer-based inference.
• Transposed Convolutions - Implements transposed convolutions to expand input data and reconstruct spatial resolution during upsampling.
• Tensor Caches - Stores reordered weights and folded scales in memory to eliminate redundant computations across hardware engines.
• Binary Footprint Reduction - Implements build-time optimizations to limit available primitives and decrease the overall size of the compiled binary.
• Thread Scaling - Distributes computational workloads across multiple CPU cores using threadpool-based threading models to maximize utilization.
• GPU Memory Optimizations - Optimizes memory throughput by managing format propagation and reordering data between CPU and GPU engines.
• Task Dependencies - Synchronizes execution between primitives using events or in-order streams to prevent data races in hardware-accelerated tasks.
• Hardware Runtime Configurations - Defines the threading model and offload runtime to match the target hardware environment.
• Inference Buffer Management - Manages the allocation and sizing of short-term memory buffers for intermediate deep learning calculations.
• Unified Memory Managers - Implements unified memory management for high-performance data transfer between host and device memory spaces.
• Micro-Kernel Compositions - Enables the composition of sequential micro-kernels to implement specialized operations with manual control over threading and blocking.
• Microkernel Execution Control - Provides manual control over threading and blocking logic on CPUs via microkernel interfaces to maximize hardware customization.
• Custom Operator APIs - Provides low-level APIs to compose sequential block-level operations and tailor threading logic for specific applications.
• Heterogeneous Hardware Runtimes - Shares data and execution contexts with external runtimes to coordinate workloads across heterogeneous CPU and GPU cores.
• Normalization and Activation Fusions - Fuses normalization, binary addition, and activation into a single operation to increase training speed and reduce memory overhead.
• Sequential Operation Fusion - Combines primitives with element-wise operations into single kernels to eliminate intermediate memory writes and reduce overhead.
• Calculation Precision Configurations - Provides settings for managing numerical precision and significant digits during mathematical computations to balance speed and accuracy.
• Element-wise Array Operations - Applies high-performance arithmetic, comparison, or selection operators between two tensors element-wise.
• Floating-Point Precision Conversions - Converts floating-point data to narrower types during computation to increase speed while maintaining original storage precision.
• Tensor Manipulations - Performs complex tensor transformations including binary operations, concatenation, reduction, and reordering of data shapes.
• Numerical Result Determinism - Guarantees that multiple executions of the same operation return bit-wise identical results for debugging and validation.
• Operation Benchmarkers - Measures the execution time of specific deep learning primitives to verify optimization gains.
• Performance Analysis - Uses performance profilers and verbose logging to analyze internal operations and identify algorithmic bottlenecks.
• Performance Profiling - Measures execution time and efficiency of primitives to identify and resolve performance bottlenecks.

Star history