NVIDIA/TensorRT
TensorRT
TensorRT ist eine Deep-Learning-Inferenz-Engine und ein Software Development Kit zur Optimierung und Bereitstellung neuronaler Netze für die Hochleistungsausführung auf NVIDIA GPUs. Es fungiert als GPU-Beschleunigungs-Framework, das Latenzzeiten reduziert und den Durchsatz für trainierte Modelle während der Produktion erhöht.
Das Toolkit importiert Modelle aus dem Open Neural Network Exchange Format und transformiert diese in optimierte Engines. Es nutzt graphbasierte Modelloptimierung, Layer-Fusion-Kernel-Generierung und präzisionsbasierte Quantisierung, um Fließkomma-Gewichte in Formate mit geringerer Präzision zu konvertieren.
Das Framework bietet Funktionen für die hardware-spezifische Engine-Serialisierung und unterstützt die Erweiterung der Inferenzfähigkeiten durch benutzerdefinierte Plugins für spezialisierte Schichten neuronaler Netze.
Features
- Model Inference Accelerators - Transforms neural networks into high-performance engines to maximize execution speed on NVIDIA GPUs.
- Cross-Format Model Importers - Imports model definitions from the ONNX format to prepare them for optimized GPU execution.
- ONNX Model Importers - Parses Open Neural Network Exchange models to build internal representations for GPU optimization.
- GPU Inference SDKs - Provides a comprehensive SDK for optimizing and deploying deep learning models on NVIDIA GPUs.
- GPU Model Deployments - Enables the deployment of optimized deep learning models on NVIDIA GPU hardware accelerators.
- GPU-Accelerated - Optimizes deep learning models for maximum throughput and low latency on GPU accelerators.
- Deep Learning - Serves as a high-performance runtime environment that executes neural networks using NVIDIA GPU acceleration.
- ONNX Engine Conversions - Converts models from the ONNX format into high-performance engines for NVIDIA GPU execution.
- ONNX Model Optimizers - Imports ONNX models and transforms them into optimized engines for faster inference.
- Hardware-Specific Model Optimizations - Compiles models into binary engines optimized for specific NVIDIA GPU architectures and memory limits.
- Model Graph Optimizers - Provides graph-level optimizations by fusing layers and removing redundant operations to improve inference performance.
- Neural Network Deployment - Provides the runtime and tools necessary to execute trained neural networks in production environments.
- Precision Quantization - Converts floating point weights to lower precision formats like FP16 or INT8 to increase throughput.
- GPU Acceleration - Provides a framework of tools to reduce latency and increase throughput for models deployed on GPUs.
- Deep Learning Acceleration - Accelerates deep learning tensor operations and matrix multiplications on NVIDIA GPU hardware.
- Custom Neural Network Layers - Allows for the implementation of specialized neural network layers via custom plugins.
- Kernel Fusion Compilers - Generates fused kernels that combine multiple neural network layers to reduce memory bandwidth overhead.
- Inference Capability Extensions - Allows adding specialized operations or layers to the runtime through custom plugin implementation.
- Custom Operator Plugins - Supports the execution of custom neural network layers via external C++ plugin implementations.
- AI & Machine Learning - High-performance inference on NVIDIA GPUs
- Parallele Verarbeitung - Hochleistungs-Inferenzbibliothek für NVIDIA-GPUs.
- Computation and Optimization - C++ library for high-performance inference on NVIDIA hardware.
- Parallel and High-Performance Computing - High-performance inference library for NVIDIA GPUs.
