NVIDIA/TensorRT
TensorRT
TensorRT es un motor de inferencia de aprendizaje profundo y un kit de desarrollo de software diseñado para optimizar y desplegar redes neuronales para una ejecución de alto rendimiento en GPUs NVIDIA. Funciona como un framework de aceleración de GPU que reduce la latencia y aumenta el rendimiento de los modelos entrenados durante el despliegue en producción.
El kit de herramientas importa modelos desde el formato Open Neural Network Exchange y los transforma en motores optimizados. Utiliza optimización de modelos basada en grafos, generación de kernels de fusión de capas y cuantización basada en precisión para convertir pesos de punto flotante a formatos de menor precisión.
El framework proporciona capacidades para la serialización de motores específicos de hardware y admite la extensión de las capacidades de inferencia a través de complementos personalizados para capas de redes neuronales especializadas.
Features
- Model Inference Accelerators - Transforms neural networks into high-performance engines to maximize execution speed on NVIDIA GPUs.
- Cross-Format Model Importers - Imports model definitions from the ONNX format to prepare them for optimized GPU execution.
- ONNX Model Importers - Parses Open Neural Network Exchange models to build internal representations for GPU optimization.
- GPU Inference SDKs - Provides a comprehensive SDK for optimizing and deploying deep learning models on NVIDIA GPUs.
- GPU Model Deployments - Enables the deployment of optimized deep learning models on NVIDIA GPU hardware accelerators.
- GPU-Accelerated - Optimizes deep learning models for maximum throughput and low latency on GPU accelerators.
- Deep Learning - Serves as a high-performance runtime environment that executes neural networks using NVIDIA GPU acceleration.
- ONNX Engine Conversions - Converts models from the ONNX format into high-performance engines for NVIDIA GPU execution.
- ONNX Model Optimizers - Imports ONNX models and transforms them into optimized engines for faster inference.
- Hardware-Specific Model Optimizations - Compiles models into binary engines optimized for specific NVIDIA GPU architectures and memory limits.
- Model Graph Optimizers - Provides graph-level optimizations by fusing layers and removing redundant operations to improve inference performance.
- Neural Network Deployment - Provides the runtime and tools necessary to execute trained neural networks in production environments.
- Precision Quantization - Converts floating point weights to lower precision formats like FP16 or INT8 to increase throughput.
- GPU Acceleration - Provides a framework of tools to reduce latency and increase throughput for models deployed on GPUs.
- Deep Learning Acceleration - Accelerates deep learning tensor operations and matrix multiplications on NVIDIA GPU hardware.
- Custom Neural Network Layers - Allows for the implementation of specialized neural network layers via custom plugins.
- Kernel Fusion Compilers - Generates fused kernels that combine multiple neural network layers to reduce memory bandwidth overhead.
- Inference Capability Extensions - Allows adding specialized operations or layers to the runtime through custom plugin implementation.
- Custom Operator Plugins - Supports the execution of custom neural network layers via external C++ plugin implementations.
- AI & Machine Learning - High-performance inference on NVIDIA GPUs
- Procesamiento paralelo - Biblioteca de inferencia de alto rendimiento para GPUs NVIDIA.
- Computation and Optimization - C++ library for high-performance inference on NVIDIA hardware.
- Parallel and High-Performance Computing - High-performance inference library for NVIDIA GPUs.
