NVIDIA/TensorRT
TensorRT
TensorRT هو محرك استنتاج للتعلم العميق ومجموعة تطوير برمجيات مصممة لتحسين ونشر الشبكات العصبية للتنفيذ عالي الأداء على وحدات معالجة الرسومات (GPUs) من NVIDIA. يعمل كإطار عمل لتسريع وحدات معالجة الرسومات يقلل من زمن الوصول ويزيد من الإنتاجية للنماذج المدربة أثناء النشر في بيئة الإنتاج.
تقوم مجموعة الأدوات باستيراد النماذج من تنسيق Open Neural Network Exchange وتحويلها إلى محركات محسنة. تستخدم تحسين النماذج القائم على الرسم البياني، وتوليد نواة دمج الطبقات، والتكميم القائم على الدقة لتحويل أوزان الفاصلة العائمة إلى تنسيقات ذات دقة أقل.
يوفر إطار العمل إمكانات لتسلسل المحرك الخاص بالأجهزة ويدعم توسيع قدرات الاستنتاج من خلال إضافات مخصصة لطبقات الشبكات العصبية المتخصصة.
Features
- Model Inference Accelerators - Transforms neural networks into high-performance engines to maximize execution speed on NVIDIA GPUs.
- Cross-Format Model Importers - Imports model definitions from the ONNX format to prepare them for optimized GPU execution.
- ONNX Model Importers - Parses Open Neural Network Exchange models to build internal representations for GPU optimization.
- GPU Inference SDKs - Provides a comprehensive SDK for optimizing and deploying deep learning models on NVIDIA GPUs.
- GPU Model Deployments - Enables the deployment of optimized deep learning models on NVIDIA GPU hardware accelerators.
- GPU-Accelerated - Optimizes deep learning models for maximum throughput and low latency on GPU accelerators.
- Deep Learning - Serves as a high-performance runtime environment that executes neural networks using NVIDIA GPU acceleration.
- ONNX Engine Conversions - Converts models from the ONNX format into high-performance engines for NVIDIA GPU execution.
- ONNX Model Optimizers - Imports ONNX models and transforms them into optimized engines for faster inference.
- Hardware-Specific Model Optimizations - Compiles models into binary engines optimized for specific NVIDIA GPU architectures and memory limits.
- Model Graph Optimizers - Provides graph-level optimizations by fusing layers and removing redundant operations to improve inference performance.
- Neural Network Deployment - Provides the runtime and tools necessary to execute trained neural networks in production environments.
- Precision Quantization - Converts floating point weights to lower precision formats like FP16 or INT8 to increase throughput.
- GPU Acceleration - Provides a framework of tools to reduce latency and increase throughput for models deployed on GPUs.
- Deep Learning Acceleration - Accelerates deep learning tensor operations and matrix multiplications on NVIDIA GPU hardware.
- Custom Neural Network Layers - Allows for the implementation of specialized neural network layers via custom plugins.
- Kernel Fusion Compilers - Generates fused kernels that combine multiple neural network layers to reduce memory bandwidth overhead.
- Inference Capability Extensions - Allows adding specialized operations or layers to the runtime through custom plugin implementation.
- Custom Operator Plugins - Supports the execution of custom neural network layers via external C++ plugin implementations.
- AI & Machine Learning - High-performance inference on NVIDIA GPUs
- المعالجة المتوازية - مكتبة استنتاج عالية الأداء لمعالجات NVIDIA الرسومية.
- Computation and Optimization - C++ library for high-performance inference on NVIDIA hardware.
- Parallel and High-Performance Computing - High-performance inference library for NVIDIA GPUs.
