NVlabs/tiny-cuda-nn
Tiny Cuda Nn
This project is a high-performance C++ and CUDA neural network library designed for fast training and inference of small networks on NVIDIA GPUs. It serves as a specialized backend for neural radiance fields and coordinate-based networks, providing a fused GPU kernel library and a hash grid encoder for transforming raw input dimensions into high-dimensional representations.
The library distinguishes itself through the use of C++ template metaprogramming and fused-kernel execution, which merge neural network layers into single GPU device functions to eliminate memory bottlenecks. It leverages tensor-core accelerated GEMM for high-throughput linear algebra and implements multiresolution hash encoding and spherical harmonic encoding to capture fine spatial and angular details.
The system covers a broad range of capabilities including 3D scene reconstruction, signed distance function implementation, and path radiance caching. It includes a comprehensive suite of training tools for weight optimization and loss calculation, as well as utilities for environment lighting approximation and material decomposition.
Low-level CUDA implementations and fast multilayer perceptrons are exposed as extensions for use within Python environments via a PyTorch C++ extension.
Features
- CUDA Accelerated Neural Networks - Ships a high-performance C++ and CUDA library for training and inference of small neural networks on NVIDIA GPUs.
- Coordinate-Based Neural Mappings - Trains neural networks that map spatial coordinates to output values for 3D scene reconstruction and signed distance functions.
- 3D Reconstruction - Optimizes topology, materials, and lighting from multi-view images to produce triangle meshes with PBR textures.
- Small MLP Inference Accelerators - Provides fused GPU kernels optimized for low-latency inference of compact multi-layer perceptrons.
- Neural Network Layer Fusers - Implements fused GPU kernels that merge neural network layers into single device functions for reduced memory traffic.
- Neural Network Layer Fusions - Provides fused GPU kernels that merge neural network layers into single device functions for high throughput.
- Just-In-Time Kernel Compilers - Compiles fully fused forward and backward kernels just-in-time to adapt to specific network architectures.
- Neural Network Kernel Compilers - Compiles fully fused forward and backward kernels at runtime for adaptive network architectures.
- JIT Kernel Fusion Accelerators - Compiles and fuses network operations at runtime to achieve 1.5x to 2.5x performance gains on compatible GPUs.
- Fully Fused MLP Training - Trains small multi-layer perceptrons with hidden layers restricted to sizes 16, 32, 64, or 128 for maximum GPU throughput.
- Network Inference Execution - Runs inference on trained small neural networks with low latency using optimized GPU kernels.
- Neural Network Training Frameworks - Trains fully fused multi-layer perceptrons on GPU with configurable input encodings, loss functions, and optimizers.
- CUTLASS MLP Training - Trains multi-layer perceptrons with arbitrary hidden and output neuron counts using CUTLASS GEMM routines.
- Spatial Coordinate Encodings - Maps input coordinates to a multi-resolution hash table for efficient feature encoding in neural fields.
- PyTorch Kernel Integrations - Calls trained PyTorch models from Python with automatic FP16 precision for fast MLP operations.
- PyTorch Bindings - Exposes low-level CUDA implementations as Python-callable PyTorch extensions for integration.
- High-Dimensional Spatial Encodings - Transforms low-dimensional inputs into high-dimensional feature vectors using multiresolution hash grids.
- Multiresolution Hash Encoders - Provides a multiresolution hash table encoder that transforms raw input coordinates into high-dimensional feature vectors.
- L1 Pixel Loss - Computes the standard L1 loss between network predictions and targets.
- Input Encoding Combinations - Combines multiple input encodings by applying a different encoding to each subset of input dimensions.
- Frequency Encodings - Transforms each input dimension into sine and cosine pairs at logarithmically spaced frequencies.
- OneBlob Encodings - Encodes each input dimension into a set of bins using a quartic kernel for accurate fitting with limited dynamic range.
- TriangleWave Encodings - Encodes each input dimension using a cheap-to-compute triangle wave at multiple frequencies.
- Loss Function Implementations - Computes the standard L2 loss between network predictions and targets.
- Adam Optimizers - Trains network parameters using the Adam optimizer with optional AdaBound generalization.
- Neural Network Optimizers - Trains networks using Adam, SGD, Novograd, or Shampoo optimizers with configurable loss functions.
- Cross-Entropy Loss Functions - Computes standard cross entropy loss for probability density function predictions.
- Stochastic Gradient Descent - Trains network parameters using standard stochastic gradient descent optimization.
- GPU-Accelerated Backends - Serves as a specialized GPU-accelerated backend for neural radiance fields with hash grid encoding.
- Material Decompositions - Separates reconstructed 3D models into spatially-varying PBR materials and HDR environment lighting.
- Multi-View Scene Reconstructions - Optimizes topology, materials, and lighting from multi-view images to produce a triangle mesh with PBR textures and environment lighting.
- Tensor Core Matrix Multiplications - Leverages NVIDIA tensor cores for high-throughput matrix multiplication in fully connected layers.
- Template Metaprogramming - Uses C++ template metaprogramming to generate specialized kernel code at compile time for optimal performance.
- Grid Backends - Supports multiple grid backends including hash tables, dense storage, and tiled storage for flexible input encoding.
- Spherical Harmonic Encoders - Encodes 3D direction vectors into spherical harmonic coefficients for frequency-space representation.
