NVIDIA/TensorRT
TensorRT
TensorRT est un moteur d'inférence d'apprentissage profond et un kit de développement logiciel conçu pour optimiser et déployer des réseaux de neurones pour une exécution haute performance sur les GPU NVIDIA. Il fonctionne comme un framework d'accélération GPU qui réduit la latence et augmente le débit des modèles entraînés lors du déploiement en production.
Le toolkit importe des modèles depuis le format Open Neural Network Exchange et les transforme en moteurs optimisés. Il utilise l'optimisation de modèles basée sur les graphes, la génération de noyaux de fusion de couches et la quantification basée sur la précision pour convertir les poids en virgule flottante en formats de précision inférieure.
Le framework fournit des capacités de sérialisation de moteur spécifique au matériel et prend en charge l'extension des capacités d'inférence via des plugins personnalisés pour des couches de réseaux de neurones spécialisées.
Features
- Model Inference Accelerators - Transforms neural networks into high-performance engines to maximize execution speed on NVIDIA GPUs.
- Cross-Format Model Importers - Imports model definitions from the ONNX format to prepare them for optimized GPU execution.
- ONNX Model Importers - Parses Open Neural Network Exchange models to build internal representations for GPU optimization.
- GPU Inference SDKs - Provides a comprehensive SDK for optimizing and deploying deep learning models on NVIDIA GPUs.
- GPU Model Deployments - Enables the deployment of optimized deep learning models on NVIDIA GPU hardware accelerators.
- GPU-Accelerated - Optimizes deep learning models for maximum throughput and low latency on GPU accelerators.
- Deep Learning - Serves as a high-performance runtime environment that executes neural networks using NVIDIA GPU acceleration.
- ONNX Engine Conversions - Converts models from the ONNX format into high-performance engines for NVIDIA GPU execution.
- ONNX Model Optimizers - Imports ONNX models and transforms them into optimized engines for faster inference.
- Hardware-Specific Model Optimizations - Compiles models into binary engines optimized for specific NVIDIA GPU architectures and memory limits.
- Model Graph Optimizers - Provides graph-level optimizations by fusing layers and removing redundant operations to improve inference performance.
- Neural Network Deployment - Provides the runtime and tools necessary to execute trained neural networks in production environments.
- Precision Quantization - Converts floating point weights to lower precision formats like FP16 or INT8 to increase throughput.
- GPU Acceleration - Provides a framework of tools to reduce latency and increase throughput for models deployed on GPUs.
- Deep Learning Acceleration - Accelerates deep learning tensor operations and matrix multiplications on NVIDIA GPU hardware.
- Custom Neural Network Layers - Allows for the implementation of specialized neural network layers via custom plugins.
- Kernel Fusion Compilers - Generates fused kernels that combine multiple neural network layers to reduce memory bandwidth overhead.
- Inference Capability Extensions - Allows adding specialized operations or layers to the runtime through custom plugin implementation.
- Custom Operator Plugins - Supports the execution of custom neural network layers via external C++ plugin implementations.
- AI & Machine Learning - High-performance inference on NVIDIA GPUs
- Traitement parallèle - Bibliothèque d'inférence haute performance pour GPU NVIDIA.
- Computation and Optimization - C++ library for high-performance inference on NVIDIA hardware.
- Parallel and High-Performance Computing - High-performance inference library for NVIDIA GPUs.
