NVIDIA/TensorRT
TensorRT
TensorRT este un motor de inferență pentru deep learning și un kit de dezvoltare software conceput pentru a optimiza și implementa rețele neuronale pentru execuție de înaltă performanță pe GPU-uri NVIDIA. Acesta funcționează ca un framework de accelerare GPU care reduce latența și crește debitul pentru modelele antrenate în timpul implementării în producție.
Toolkit-ul importă modele din formatul Open Neural Network Exchange și le transformă în motoare optimizate. Utilizează optimizarea modelelor bazată pe grafuri, generarea de kernel-uri prin fuziunea straturilor și cuantizarea bazată pe precizie pentru a converti ponderile în virgulă mobilă în formate cu precizie mai mică.
Framework-ul oferă capabilități pentru serializarea motoarelor specifice hardware-ului și suportă extinderea capabilităților de inferență prin plugin-uri personalizate pentru straturi specializate de rețele neuronale.
Features
- Model Inference Accelerators - Transforms neural networks into high-performance engines to maximize execution speed on NVIDIA GPUs.
- Cross-Format Model Importers - Imports model definitions from the ONNX format to prepare them for optimized GPU execution.
- ONNX Model Importers - Parses Open Neural Network Exchange models to build internal representations for GPU optimization.
- GPU Inference SDKs - Provides a comprehensive SDK for optimizing and deploying deep learning models on NVIDIA GPUs.
- GPU Model Deployments - Enables the deployment of optimized deep learning models on NVIDIA GPU hardware accelerators.
- GPU-Accelerated - Optimizes deep learning models for maximum throughput and low latency on GPU accelerators.
- Deep Learning - Serves as a high-performance runtime environment that executes neural networks using NVIDIA GPU acceleration.
- ONNX Engine Conversions - Converts models from the ONNX format into high-performance engines for NVIDIA GPU execution.
- ONNX Model Optimizers - Imports ONNX models and transforms them into optimized engines for faster inference.
- Hardware-Specific Model Optimizations - Compiles models into binary engines optimized for specific NVIDIA GPU architectures and memory limits.
- Model Graph Optimizers - Provides graph-level optimizations by fusing layers and removing redundant operations to improve inference performance.
- Neural Network Deployment - Provides the runtime and tools necessary to execute trained neural networks in production environments.
- Precision Quantization - Converts floating point weights to lower precision formats like FP16 or INT8 to increase throughput.
- GPU Acceleration - Provides a framework of tools to reduce latency and increase throughput for models deployed on GPUs.
- Deep Learning Acceleration - Accelerates deep learning tensor operations and matrix multiplications on NVIDIA GPU hardware.
- Custom Neural Network Layers - Allows for the implementation of specialized neural network layers via custom plugins.
- Kernel Fusion Compilers - Generates fused kernels that combine multiple neural network layers to reduce memory bandwidth overhead.
- Inference Capability Extensions - Allows adding specialized operations or layers to the runtime through custom plugin implementation.
- Custom Operator Plugins - Supports the execution of custom neural network layers via external C++ plugin implementations.
- AI & Machine Learning - High-performance inference on NVIDIA GPUs
- Procesare paralelă - Bibliotecă de inferență de înaltă performanță pentru GPU-uri NVIDIA.
- Computation and Optimization - C++ library for high-performance inference on NVIDIA hardware.
- Parallel and High-Performance Computing - High-performance inference library for NVIDIA GPUs.
