dusty-nv/jetson-inference

Features

• Deep Learning Inference Engines - Provides a high-performance runtime engine for executing deep learning models on embedded GPU hardware via TensorRT.
• GPU Accelerated Computer Vision - Uses GPU-optimized libraries to accelerate real-time image processing, depth estimation, and pose tracking.
• Inference Execution - Executes optimized deep learning models on specialized GPU hardware to produce fast, accurate predictions.
• Computer Vision Platforms - Provides a comprehensive environment for developing and deploying real-time computer vision applications on embedded hardware.
• Edge AI Model Deployment - Running optimized deep learning models on embedded GPU hardware for real-time computer vision and robotics.
• GPU Performance Profilers - Analyzes and debugs GPU-accelerated workloads to optimize AI, graphics, and compute performance.
• AI Hosting Platforms - Deploys pretrained or customized AI models as GPU-accelerated containers using industry-standard APIs.
• AI Model Integrations - Incorporates curated SDKs and pre-trained models to accelerate the addition of AI capabilities to applications.
• AI Workload Orchestration - Provides specialized libraries for AI, mathematics, and data science to accelerate complex computational workloads.
• Cross-Format Model Importers - Converts models from PyTorch, Hugging Face, and ONNX formats into high-performance inference engines.
• Computer Vision Pipelines - Constructs real-time data processing workflows to streamline data movement from sensors to AI inference.
• Computer Vision Libraries - Provides a library for executing optimized neural network primitives and computer vision tasks on edge devices.
• Vision Pipeline Orchestrators - Develops streaming pipelines that ingest videos and preprocess frames for optimized vision AI models.
• Object Detection - Localizes objects using 2D bounding boxes to provide a front end for pose estimation.
• Gesture Recognition Libraries - Identifies common hand gestures such as waving or thumbs-up from real-time visual streams.
• Python Bindings - Provides low-level Python bindings and core runtime functionalities for direct CUDA platform interaction.
• GPU-Accelerated Inference - Accelerates the inference phase of machine learning models for image, video, and audio data on GPUs.
• Robotics Pipeline Acceleration - Optimizes data transport and GPU resource utilization across robotics graphs using hardware-accelerated modules.
• GPU Kernel Implementations - Executes asynchronous, fine-grained data movements initiated directly by GPU threads to eliminate CPU overhead.
• Image Classification - Implements deep learning models like ResNet and VGG to identify objects and labels within images.
• Multi-Stage Inference Pipelines - Links multiple models and preprocessing steps into a single execution graph for complex vision and audio workflows.
• Inference API Servers - Exposes guardrailed inference through a standalone API server, Docker containers, or production microservices.
• Inference Optimizations - Transforms neural network models to reduce latency and increase throughput for production deployment.
• Model Compilation Memory Optimization - Manages memory allocation to enable the deployment of large foundation models on resource-constrained edge devices.
• Model Performance Optimization - Uses accelerated engines to reduce response latency and increase throughput for specific GPU hardware.
• Model Quantization - Converts high-precision checkpoints into quantized engines to reduce VRAM usage and increase speed.
• Primitive Accelerators - Executes highly tuned GPU-accelerated routines for convolution, pooling, normalization, and activation layers.
• Pose Estimation - Recognizes and tracks anatomical points on the human body within images or video streams.
• Weight Quantization - Reduces model precision using FP8 and INT4 formats to lower memory usage and accelerate execution.
• Neural Network Compression - Applies quantization, pruning, sparsity, and distillation to reduce model size and increase execution efficiency.
• Robotics Perception Acceleration - Runs hardware-accelerated packages for high-performance perception and localization in robotic systems.
• Concurrent Model Execution - Executes multiple deep learning inference streams simultaneously on auto-grade silicon for real-time tasks.
• Ensemble Inference Pipelines - Links multiple models and pre- or post-processing steps into a single ensemble to handle complex inference workflows.
• Video Analytics Pipelines - Provides pipelines for real-time object detection, tracking, and segmentation on live video streams.
• Action Recognition - Analyzes video sequences to identify and classify specific human activities or behaviors over time.
• Zero-Copy Framework Integrations - Shares data with deep learning frameworks via zero-copy interfaces to eliminate expensive memory transfers.
• Neural Network Acceleration - NVIDIA provides optimized CUDA kernels for triangle attention and triangle multiplication to speed up processing of 3D data.
• Device Management and Deployment - Deploys, scales, and updates AI applications and system software over the air across a fleet of edge devices.
• GPU Acceleration - Provides a comprehensive suite of compilers and runtime libraries for building high-performance GPU-accelerated applications.
• Deep Learning Acceleration - NVIDIA runs highly tuned GPU-accelerated routines for convolution, attention, matmul, pooling, and normalization.
• ROS Libraries and Tools - Provides inference nodes to incorporate deep learning capabilities into Robot Operating System (ROS/ROS2) environments.
• Sensor - NVIDIA handles high-bandwidth data from diverse sensors over Ethernet to enable real-time AI processing.
• Sensor - Cleans, filters, and transforms raw sensor data into structured formats using GPU-accelerated libraries.
• GPU-Accelerated Data Streams - Implements high-throughput, low-latency data streaming to share GPU data between systems.
• Stream Analytics Processing - Analyzes concurrent video, audio, and image data using a streaming analytics toolkit for real-time understanding.
• Image Buffer Sharing - Implements zero-copy data transport to move camera frames directly into GPU memory without duplication.
• Shared Memory Transports - Implements zero-copy memory transport to share data buffers between libraries without expensive CPU-to-GPU transfers.
• Microservice Infrastructure - Packages high-performance AI inference as secure, reliable microservices for deployment across clouds and data centers.
• Hardware-Accelerated Decoders - Utilizes on-chip codecs to decompress video and image streams directly into GPU memory for real-time processing.
• Real-Time Video Analysis - Builds vision applications that perform real-time video analytics with accelerated inference and object tracking.
• Hardware-Accelerated Video Pipelines - Implements a framework for hardware-accelerated decoding, encoding, and processing of video and audio streams.
• Multimedia Processing - Provides low-level hardware access to cameras and video processing for high-performance multimedia pipelines.
• Stream Decoding - NVIDIA decompresses video from multiple popular codecs using on-chip hardware acceleration.
• Stream Encoding - NVIDIA compresses video into various formats including H.264, HEVC, and AV1 using dedicated hardware.
• Video Encoding and Decoding - NVIDIA accelerates video encoding and decoding using hardware-specific APIs on Windows and Linux.
• Edge AI Perception Toolkits - Provides tools for integrating camera sensors and AI models into robotics and autonomous systems.
• Sensor Processing - Builds high-performance sensor-processing pipelines using zero-copy data transport directly into GPU memory.
• GPU Computations - Leverages parallel processing power on GPUs to execute computationally intensive tasks through Python applications.
• Modular Camera Backends - Decouples sensor ingestion from inference logic to support diverse camera inputs across different hardware platforms.
• GPU Memory Orchestration - Manages data transfers between GPUs using CPU-based operations and CUDA streams.
• Unified Memory Managers - Manages low-level memory allocation and access between host and device to simplify GPU-accelerated development.
• Zero-Copy Mechanisms - Uses columnar memory formats and zero-copy interfaces to minimize data transfer overhead between CPU and GPU.
• Inference Performance Monitoring - Provides detailed observability metrics and Helm charts to monitor and scale AI inference microservices.
• 3D Pose Reconstruction - Tracks skeletal movement from a single camera to reconstruct full-body 3D animations without markers.
• AI Model Orchestration - Manages and executes both local and cloud AI models on edge devices for autonomous, multimodal applications.
• Throughput Optimizations - Increases inference throughput using custom attention kernels, in-flight batching, and paged KV caching.
• Cross-Camera Tracking - Maintains unique object identities across a network of multiple cameras to handle occlusions.
• Object Pose Estimations - Tracks the 6D pose of novel objects using foundation models to determine exact position and orientation.
• Monocular Depth Estimators - Predicts spatial depth from a single camera lens using monocular depth estimation algorithms.
• Depth Estimation - Calculates distance and spatial geometry using both monocular and stereo depth estimation models.
• Image Segmentation - Implements pixel-level classification to define precise shapes and boundaries of objects in images.
• Inference Speed Profiling - Profiles model performance and analyzes execution timing to tune inference speed and efficiency.
• Driving Scenario Generation - Produces photorealistic world variations from text prompts and spatial controls to expand driving datasets.
• Generative AI Model Serving - Distributes generative AI inference workloads across GPU fleets using intelligent resource scheduling and request routing.
• Generative AI Development - Provides an ecosystem of tools for developing conversational agents, copilots, and generative AI search engines.
• GPU Acceleration - NVIDIA executes existing scikit-learn, UMAP, or HDBSCAN code on GPUs without requiring modifications to the source code.
• Tile-Based Kernel Authoring - Implements a tile programming model in C++ and Python to manage high-performance data movement across GPU threads.
• Multi-Node Inference Scaling - NVIDIA deploys large models across multiple GPUs and nodes using pipeline parallelism to handle models exceeding single-GPU memory.
• Inference Acceleration - Reduces latency and increases throughput for large language model execution using a simplified API.
• Inference Scaling Frameworks - NVIDIA integrates with Kubernetes and cloud orchestration environments to deploy and scale deep learning models across clusters.
• Just-In-Time Kernel Compilers - Translates Python functions into optimized CUDA kernels at runtime for fine-grained thread control.
• Large Language Model Serving - Provides high-speed inference and serving for large language models and vision language models.
• Large-Scale Training Frameworks - Implements data and model parallelism for foundational scale models using a GPU-accelerated distributed framework.
• Vision AI Agents - Develops intelligent vision applications that process visual data to automate tasks or monitor environments.
• High-Performance AI Inference - Executes large language models with a modular runtime to maximize throughput on GPU hardware.
• GPU Training Accelerators - Executes collective communication operations to distribute large models across multiple GPUs for faster training.
• Mixed Precision Training - Reduces training time using multi-GPU distribution and mixed-precision floating-point computations.
• Vision Model Fine-Tuning - Adapts pre-trained vision backbones and foundation models using domain-specific data and natural language prompts.
• LLM Serving Architectures - Implements high-performance serving architectures for large and vision language models.
• Generative AI Models - NVIDIA runs large language models and vision transformers on embedded hardware to enable real-time AI in robotics and computer vision.
• Data Preprocessing - Decodes and augments images, videos, and speech in parallel with training to eliminate loading bottlenecks.
• Automated Image Labeling - Automatically generates object detection and segmentation masks using AI-driven prompts and descriptors.
• Multi-Framework Model Serving - Serves models from multiple frameworks across diverse hardware accelerators and CPUs using optimized configurations.
• Multi-Physics Simulations - Calculates multi-physics behaviors for robotics and digital twins using GPU-accelerated engines.
• Multimodal Analysis Tools - Combines image and video data with text prompts to perform feature extraction and segmentation.
• Neural Network Design Frameworks - Provides an integrated environment for the structural design and development of deep neural networks for inference.
• Real-Time Speech Processing - Develops customized, real-time speech applications using GPU-accelerated processing pipelines.
• Multi-Camera Tiled Rendering - Consolidates multi-camera input into a single image via tiled rendering for real-time agent data.
• Speech-to-Text Conversions - Transcribes spoken language into text with multi-language support and optimized memory footprints for on-device use.
• Standardized AI Component Abstractions - Executes deep learning workloads using standardized programming models to ensure portability between cloud and embedded hardware.
• Tensor Data Representations - Converts diverse 3D and multimedia formats into a consistent tensor representation for AI training and inference.
• Autonomous Vehicle Dataset Curation - Scales the labeling and curation of autonomous vehicle datasets using integrated cloud hardware and enterprise software.
• Training Data Generation - Produces augmented datasets by randomizing scene attributes like lighting and color to improve AI model robustness.
• Transfer Learning - Adapts pre-trained models to specific platforms to optimize inference throughput.
• Agentic Visual Reasoning - Builds intelligent systems that utilize computer vision and real-time visual reasoning to interact with the physical world.
• Synthetic Video Generators - Generates synthetic single and multiview videos based on vehicle data to accelerate training scenarios.
• Video Object Tracking - Follows objects across sequential video frames using optical flow to optimize GPU usage.
• 3D Reconstruction - Converts RGB-D or lidar data into dense 3D maps and temporal costmaps for navigation.
• Autonomous Driving - Combines reconstructed scenes with traffic and policy models for scalable closed-loop testing of self-driving systems.
• Large Language Model Deployments - Hosts a wide variety of large language models via standardized microservices.
• Model Evaluation and Benchmarking - Runs model benchmarks across local machines, HPC clusters, or cloud platforms using a unified interface.
• Production Traffic Scaling - NVIDIA serves optimized models using dynamic batching, concurrent execution, and model ensembling to handle production traffic.
• Robotics Simulators - Creates physically based virtual environments for robotics testing using rigid body and vehicle dynamics.
• Simulation Environments - Reconstructs real-world data into interactive simulations to test autonomous driving workflows.
• Synthetic Data Generation - Generates synthetic images and videos to expand training datasets and enhance model robustness.
• World Models & Simulation - Generates high-fidelity 3D environments and sensor data to test autonomous systems against rare environmental conditions.
• Robotic Sensor Simulation - Simulates perception hardware output, such as lidar and depth cameras, using GPU-accelerated rendering.
• Physics Simulation - Provides a GPU-accelerated physics engine to calculate interactions for robotic systems.
• Robotics Simulators - Provides virtual environments to train and validate robotic behaviors before deployment to physical hardware.
• Multi-Stage Matrix Optimization - Executes multi-stage matrix-matrix multiplications with fusion and tuning to maximize hardware performance.
• Collective GPU Communication - NVIDIA executes collective communication routines like all-reduce and broadcast to share data across multiple GPUs and nodes.
• Storage Throughput Optimizers - NVIDIA bypasses CPU bounce buffers to move data directly between storage and GPU memory.
• Dataset Preparation Tools - Ingests and converts raw data into optimized formats using pipeline management and automated labeling.
• GPUDirect Storage - NVIDIA moves sensor data directly into GPU memory using high-speed capture cards to minimize ingestion latency.
• Model-Assisted Labelers - Runs deep learning models to automatically label datasets with GPU-accelerated pre- and post-processing.
• Model Weight Conversions - Translates model weights between different formats to ensure interoperability between training frameworks and inference engines.
• GPU State Inspection - NVIDIA controls execution via breakpoints and single-stepping to inspect variables, registers, and GPU state.
• Robotics System Integrations - Supports custom ROS2 messages and URDF formats to enable standalone scripting and manual control of simulations.
• Automotive AI Deployment - Provides a full-stack platform to build and run scalable, real-time AI applications for automotive production.
• AI Deployment Containers - Runs specialized AI functions using user-provided containers, models, and Helm charts.
• LLM Inference Optimization - Accelerates large language models through prefix caching, key-value caching, and disaggregated serving.
• Background Removal Tools - Separates primary subjects from their background for isolation or replacement.
• Cloud Native Development Tools - Employs containers, Kubernetes, and microservices to create scalable AI applications bridging cloud and edge.
• Cloud Native Infrastructure - Uses containerized development and Kubernetes to scale edge AI within cloud-native infrastructure.
• Cloud Native GPU Orchestration - Scales compute workloads across on-premises, private, and public cloud resource clusters using GPU orchestration.
• Deployment Orchestration - Standardizes the training and deployment workflow across edge and cloud environments with automated tuning.
• Media Processing Scaling - Scales image and signal processing workloads across multiple GPUs to increase throughput.
• GPU Container Toolkits - Configures container runtimes to enable hardware-accelerated applications to run inside portable containers.
• Inference Engine Compilers - Creates lightweight, cross-OS and cross-GPU portable inference engines directly on target hardware.
• GPU Resource Automation - Manages the software required to expose GPUs on Kubernetes to improve performance and utilization.
• Kubernetes Deployment Management - Coordinates the startup ordering and scaling of interdependent inference components on Kubernetes.
• Model Conversion - Parses models from PyTorch, Hugging Face, and ONNX to generate optimized inference engines.
• 3D Rendering Engines - Uses GPU-accelerated APIs to perform high-performance 3D rendering and UI display.
• Hardware-Accelerated Ray Tracing - Implements a flexible pipeline for ray generation, intersection, and shading to optimize GPU ray tracing.
• Volumetric Ray Tracing - Uses hierarchical algorithms to perform fast ray tracing for city-scale neural radiance fields.
• Volumetric Rendering Engines - Accelerates the rendering of sparse volumetric data structures for real-time visualization of complex effects.
• Image Processing - Applies rectification, color correction, filtering, and feature extraction algorithms to optimize image data.
• Custom Sensor Data Pipelines - Constructs flexible processing graphs using custom operators to transform audio, image, and video data.
• Image Processing - Performs high-performance image processing and transformations directly on the GPU.
• Computer Vision Operator Acceleration - Executes a specialized set of high-performance computer vision operators on the GPU to reduce processing costs.
• Video Object Segmentations - Runs models on live video feeds to isolate specific objects using real-time query points.
• Video Dataset Processing - Processes video content using GPU-accelerated pipelines for splitting and sharding large files into training datasets.
• Motion Vector Calculation - Calculates relative pixel motion between frames using dedicated GPU hardware to track object movement.
• Real-Time Video Filtering - NVIDIA accelerates video processing for effects including AI green screens, background blur, and webcam denoising.
• Stereo Vision Reconstruction - Generates depth maps using stereo matching with zero-shot generalization for unfamiliar scenes.
• Hardware-in-the-Loop Simulators - Tests and verifies trained robot behaviors in high-fidelity physical environments before hardware deployment.
• Robotics And Autonomous Systems - Provides tools for building robotic systems including motion, perception, and autonomous navigation.
• SLAM Algorithms - Implements high-performance visual SLAM to track robot position and map environments in real-time.
• Real-Time Sensor Fusion - Processes multimodal data from images, video, and lidar to extract real-time environmental metadata.
• Vehicle Egomotion Tracking - Predicts a vehicle's pose by applying motion models to odometry and IMU measurements.
• GPU Memory Diagnostics - NVIDIA identifies memory access violations and detects precise exceptions using integrated memory checking tools.
• GPU Shared Memory Race Detection - Detects hazardous data access patterns where multiple threads access the same shared memory location.
• Uninitialized Memory Detectors - Flags instances where device global memory is read before it has been initialized.
• Remote GPU Memory Access - NVIDIA moves data between local or remote storage and GPU memory using a direct-memory access engine to bypass the CPU.
• Kernel Fusion Operations - NVIDIA combines multiple memory-bound and compute-bound operations into single kernels to reduce memory overhead.
• Signal Processing - Executes GPU-accelerated primitives for color conversion, filtering, and geometry transforms.
• In-Kernel Execution - Performs linear algebra operations directly on the device side within CUDA kernels to reduce latency.
• Linear Algebra - Performs vector and matrix calculations using hardware acceleration for dense linear algebra workloads.
• Distributed NumPy Workflows - Executes NumPy API operations across multiple GPUs and nodes to handle large-scale numerical computing.
• Application Performance Optimization - Analyzes execution traces and hardware metrics to identify bottlenecks and increase GPU code efficiency.
• GPU API Call Tracing - Registers callbacks for specific CUDA Runtime and Driver API calls to monitor entry and exit points.
• Distributed Monitoring Tools - Profiles communication patterns and reliability to debug multi-node scaling across distributed systems.
• GPU Profilers - Captures detailed logs of GPU kernel executions and memory operations with normalized timestamps.
• Hardware Monitoring Tools - Reports real-time telemetry including GPU utilization, temperatures, and power draw.
• Memory Leak Detection - Identifies out-of-bounds accesses, misaligned memory reads, and memory leaks during runtime.
• Mobile and Embedded AI - Deep learning inference tutorials and tools for NVIDIA Jetson.

Star history