OAID/Tengine
Tengine
Tengine is a suite of tooling and a lightweight execution engine designed for running deep learning models on constrained embedded hardware. It provides an infrastructure for converting neural network models, quantizing weights, optimizing operator kernels, and benchmarking inference performance across CPU, GPU, and NPU units.
The project features an automated operator kernel optimizer to generate high-efficiency kernels and a model quantization tool that reduces precision to integer formats to lower memory usage. It includes a dedicated hardware benchmarking tool to evaluate the execution speed and efficiency of neural network architectures on embedded devices.
The system covers model format conversion into a hardware-agnostic internal representation, modular operator dispatch, and multi-backend execution. These capabilities allow for the transformation of external neural network definitions into a compatible runtime format for embedded deployment.
Features
- On-Device Inference Executions - Runs deep learning models on constrained embedded hardware by utilizing CPUs, GPUs, and NPUs for efficient execution.
- Deep Learning Inference Engines - Provides a lightweight execution engine for running deep learning models on constrained embedded hardware across CPU, GPU, and NPU units.
- Hardware Dispatchers - Routes neural network operations to specific hardware-optimized kernels based on the available compute unit.
- Intermediate Model Representations - Uses a standardized intermediate representation to decouple model conversion from device-specific execution.
- Model Quantization Tools - Reduces model precision to integer formats to lower memory usage and increase execution speed.
- Embedded Quantization Toolsets - Reduces model precision to integer formats to lower memory consumption and increase processing speed on embedded devices.
- Model Format Converters - Transforms trained neural network models into a compatible internal format for embedded deployment.
- Weight Quantization - Reduces model precision from floating point to integers to lower memory footprint and accelerate embedded execution.
- Multi-Backend Execution - Executes model graphs across CPU, GPU, and NPU units through a common abstraction layer.
- Model Conversion - Transforms neural network models into specialized internal formats compatible with limited hardware environments.
- Model Format Optimizers - Transforms neural network models into a compatible internal format using local binaries or browser-based tools.
- Neural Network Instruction Execution - Supports executing deep learning models across various compute units including the CPU, GPU, and NPU on embedded devices.
- Operator Kernel Implementations - Generates high-efficiency operator kernels to improve execution speed and resource utilization for deep learning tasks.
- Automated Kernel Generators - Provides an automated optimizer to generate high-efficiency operator kernels for targeted hardware architectures.
- Cross-Platform Model Execution - Transforms external neural network definitions into a compatible runtime format for cross-platform execution.
- Architecture Benchmarking Tools - Evaluates the execution speed of neural network architectures on embedded devices through comparative benchmarking.
- AI Inference Benchmarks - Measures the execution speed and efficiency of neural network architectures on specific embedded compute units.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
