dusty-nv/jetson-containers
Jetson Containers
Jetson Containers es un sistema de gestión de contenedores que construye y ejecuta imágenes de Docker aceleradas por GPU para cargas de trabajo de aprendizaje automático en hardware de borde ARM64. Funciona como un orquestador de contenedores CUDA, detectando automáticamente la versión del kit de herramientas CUDA del host y las capacidades de la GPU para garantizar la compatibilidad del contenedor en tiempo de ejecución, mientras selecciona la imagen de contenedor correcta haciendo coincidir la versión de JetPack o L4T del host en el momento del lanzamiento.
El proyecto ofrece contenedores preconfigurados para ejecutar modelos de lenguaje grandes cuantizados y tuberías de generación aumentada por recuperación optimizadas para dispositivos de borde, junto con contenedores integrados de ROS y marcos de IA para desplegar agentes autónomos y procesamiento multimodal. Su sistema de construcción en capas modular ensambla imágenes de Docker a partir de capas reutilizables preconstruidas, compilando marcos de IA/ML desde la fuente para optimizarlos para arquitecturas de GPU de borde específicas y versiones de CUDA, con almacenamiento en caché de ruedas local para acelerar las construcciones posteriores.
La plataforma proporciona contenedores de Docker preconstruidos con versiones aceleradas por GPU de PyTorch, TensorFlow, JAX y ONNX Runtime para plataformas Jetson, lo que admite capacidades como la ejecución de LLM, modelos de voz, modelos de visión-lenguaje y traducción automática neuronal en hardware de borde. También permite construir contenedores personalizados con paquetes de IA acelerados por GPU, ejecutar contenedores de Triton Inference Server y Transformer Engine, y acelerar los flujos de trabajo de ciencia de datos con bibliotecas RAPIDS.
Features
- Edge AI Container Orchestrators - Orchestrates GPU-accelerated Docker containers for machine learning workloads on ARM64 edge hardware.
- Edge AI Runtimes - Provides pre-built Docker containers with GPU-accelerated AI frameworks for Jetson platforms.
- Edge AI Model Deployment - Deploys GPU-accelerated deep learning frameworks and inference engines on edge hardware.
- Edge Deployment Platforms - Provides optimized containers for executing quantized LLMs and RAG pipelines on edge devices.
- Quantized LLM Deployments - Executes quantized large language models and RAG pipelines optimized for edge GPUs.
- Multimodal Processing - Accept and process images alongside text in a single conversation turn for visual reasoning.
- Edge Device Installations - Installs PyTorch with CUDA acceleration for AI/ML workloads on edge devices.
- GPU-Optimized Source Compilations - Compiles AI/ML frameworks from source to optimize them for specific edge GPU architectures and CUDA versions.
- CUDA Runtime Detections - Detects the host's CUDA toolkit version and GPU capabilities to ensure container compatibility at runtime.
- CUDA-Aware Container Builders - Builds containers with automated CUDA detection and source-based compilation for edge GPUs.
- Host-Version-Based Image Selectors - Select a container image that matches the host's JetPack or L4T version for automatic compatibility.
- Platform-Version-Matched Container Launchers - Launch a container image that matches the host platform version, pulling or building it automatically.
- Edge Device Image Builders - Ships a build system that assembles Docker images with pre-configured AI/ML packages for edge devices.
- GPU-Accelerated Containers - Runs machine learning models with GPU acceleration on ARM64 edge hardware using pre-built Docker containers.
- CUDA Compatibility Resolution - Automatically detects host CUDA version and GPU capabilities to select compatible container images.
- System-Aware Image Resolution - Selects the correct container image by matching the host's JetPack or L4T version at launch time.
- Edge AI Integrations - Ships integrated ROS and AI framework containers for deploying autonomous agents and multimodal processing on edge hardware.
- Generative AI Edge Stacks - Delivers pre-configured containers for quantized LLMs and RAG pipelines tailored to low-power ARM64 devices.
- AI Package Build Systems - Assembles custom Docker images by combining pre-built GPU-accelerated AI/ML packages with source compilation.
- Reusable Build Layers - Assembles Docker images from reusable, pre-built layers to minimize rebuild time and maximize composability.
- Streaming Chat Responses - Send generated text output incrementally as it is produced for real-time user interaction.
- ARM SBSA CUDA Runtimes - Run the same server-class CUDA toolkit on embedded edge devices starting with platform version 7.2.
- Audio Processing - Convert and analyze audio data for speech and sound processing pipelines.
- JAX Runtimes - Runs JAX machine learning workloads with CUDA support inside containers on edge devices.
- Triton - Ships pre-built containers for running NVIDIA Triton Inference Server on edge devices.
- xLSTM Models - Run xLSTM-based language models on edge hardware using pre-built containers with GPU acceleration.
- GPU-Accelerated Training - Provides cuML-based GPU-accelerated machine learning training within containers.
- Package Installers - Install Mamba-based AI model packages for GPU-accelerated inference on edge devices.
- ONNX Runtime Inference - Run ONNX Runtime inference containers pre-built for edge devices with GPU acceleration.
- Neural Machine Translation - Run optimized transformer models for fast, GPU-accelerated translation on edge devices.
- Edge Speech Model Runtimes - Load and execute speech recognition and text-to-speech models optimized for edge AI devices.
- Vision-Language Models - Load and execute multimodal vision-language models that process both images and text on edge hardware.
- Whisper-Based Engines - Run Whisper speech-to-text models efficiently on edge hardware using a CTranslate2 backend.
- Conversational Agents - Build interactive agents that combine chat, voice, and web interfaces for multimodal conversations.
- Conversation History Stores - Store and retrieve conversation context across multiple turns for coherent dialogue.
- CUDA Version Rebuilders - Recompile the entire dependency chain for a specified CUDA toolkit version, caching wheels to speed up subsequent builds.
- TensorFlow Container Builds - Build a Docker container with a specific TensorFlow version and its GPU-accelerated dependencies for Jetson platforms.
- Transformer Engine Containers - Provides pre-built containers for running NVIDIA Transformer Engine on edge devices.
- TensorFlow Container Runs - Launch a pre-built or auto-built TensorFlow container with GPU support, device detection, and data volume mounts.
- Tool Use and Function Calling - Invoke predefined functions during text generation to automate tool use and external actions.
- Image Processing and Manipulation - Read, convert, and prepare image data for model inference on GPU.
- Containerized Robotics Stacks - Deploys ROS integrated with AI frameworks in containers for autonomous agents on edge hardware.
- RAPIDS Libraries - Delivers RAPIDS libraries integrated into containers for GPU-accelerated data science on edge devices.
- Sistemas embebidos - Machine learning container images for embedded hardware.
