OAID/Tengine
Tengine
Tengine este o suită de instrumente și un motor de execuție ușor, conceput pentru rularea modelelor de deep learning pe hardware embedded cu resurse limitate. Acesta oferă o infrastructură pentru conversia modelelor de rețele neuronale, cuantizarea ponderilor, optimizarea kernel-urilor operatorilor și benchmarking-ul performanței de inferență pe unități CPU, GPU și NPU.
Proiectul dispune de un optimizator automat de kernel-uri pentru operatori pentru a genera kernel-uri de înaltă eficiență și un instrument de cuantizare a modelelor care reduce precizia la formate întregi pentru a scădea utilizarea memoriei. Include un instrument dedicat de benchmarking hardware pentru a evalua viteza de execuție și eficiența arhitecturilor de rețele neuronale pe dispozitive embedded.
Sistemul acoperă conversia formatelor de model într-o reprezentare internă hardware-agnostică, dispatch-ul modular al operatorilor și execuția multi-backend. Aceste capabilități permit transformarea definițiilor externe de rețele neuronale într-un format de runtime compatibil pentru deployment-ul embedded.
Features
- On-Device Inference Executions - Runs deep learning models on constrained embedded hardware by utilizing CPUs, GPUs, and NPUs for efficient execution.
- Deep Learning Inference Engines - Provides a lightweight execution engine for running deep learning models on constrained embedded hardware across CPU, GPU, and NPU units.
- Hardware Dispatchers - Routes neural network operations to specific hardware-optimized kernels based on the available compute unit.
- Intermediate Model Representations - Uses a standardized intermediate representation to decouple model conversion from device-specific execution.
- Model Quantization Tools - Reduces model precision to integer formats to lower memory usage and increase execution speed.
- Embedded Quantization Toolsets - Reduces model precision to integer formats to lower memory consumption and increase processing speed on embedded devices.
- Model Format Converters - Transforms trained neural network models into a compatible internal format for embedded deployment.
- Weight Quantization - Reduces model precision from floating point to integers to lower memory footprint and accelerate embedded execution.
- Multi-Backend Execution - Executes model graphs across CPU, GPU, and NPU units through a common abstraction layer.
- Model Conversion - Transforms neural network models into specialized internal formats compatible with limited hardware environments.
- Model Format Optimizers - Transforms neural network models into a compatible internal format using local binaries or browser-based tools.
- Neural Network Instruction Execution - Supports executing deep learning models across various compute units including the CPU, GPU, and NPU on embedded devices.
- Operator Kernel Implementations - Generates high-efficiency operator kernels to improve execution speed and resource utilization for deep learning tasks.
- Automated Kernel Generators - Provides an automated optimizer to generate high-efficiency operator kernels for targeted hardware architectures.
- Cross-Platform Model Execution - Transforms external neural network definitions into a compatible runtime format for cross-platform execution.
- Architecture Benchmarking Tools - Evaluates the execution speed of neural network architectures on embedded devices through comparative benchmarking.
- AI Inference Benchmarks - Measures the execution speed and efficiency of neural network architectures on specific embedded compute units.
