NVIDIA/warp
Warp
Warp is a Python framework that JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation, with built-in automatic differentiation and multi-framework array interoperability. At its core, it provides a GPU kernel compilation system that enables writing and executing custom GPU kernels directly from Python, while supporting automatic gradient computation through those kernels for integration with machine learning pipelines. The framework also includes tile-based cooperative computing, where thread blocks partition into tiles for shared-memory and tensor-core operations, and a block-sparse matrix engine for GPU-accelerated linear algebra.
What distinguishes Warp is its deep interoperability with major deep learning frameworks. It exchanges GPU arrays with PyTorch, JAX, Paddle, and NumPy via DLPack and CUDA array interfaces without copying data, and can wrap Warp kernels as callable primitives inside JAX JIT-compiled functions or PyTorch autograd graphs. The framework supports CUDA graph capture and replay for low-overhead repeated execution, multi-device orchestration across multiple GPUs, and spatial acceleration structures like BVH and hash grids for collision detection and ray casting. It also provides debugging and diagnostics tools including kernel code stepping, assertion checking, and integration with CUDA Compute Sanitizer.
The framework covers GPU-accelerated physics simulation, finite element method on GPU, geometry querying on meshes and bounding volume hierarchies, and hardware-accelerated texture sampling. It includes mathematical operations on scalars, vectors, matrices, quaternions, and spatial transforms, as well as random number generation and volume data handling. Warp can be installed via conda, built from source, or integrated as an Omniverse extension, and supports Docker container execution with the NVIDIA Container Toolkit.
Features
- Python GPU Kernels - JIT-compiles Python functions into CUDA kernels for GPU-accelerated parallel computation.
- Automatic Differentiation - Wires custom vector-Jacobian products around Warp kernels for automatic differentiation in JAX.
- Asynchronous Kernel Launchers - Schedules GPU kernel execution without blocking the host thread, enabling concurrent CPU and GPU work.
- Custom Autograd Functions - Wraps Warp kernel launches inside custom autograd functions for gradient flow through both frameworks.
- PyTorch Tensor Interoperabilities - Converts Warp arrays to and from PyTorch tensors without copying data.
- DLPack Array Exchanges - Imports and exports arrays with any DLPack-compatible framework without copying data.
- Cooperative Tile Operations - Performs cooperative parallel operations on GPU thread-block tiles including sorting, scanning, and matrix multiplication.
- Cooperative Tile Processors - Performs cooperative parallel operations on thread-block tiles including matrix multiply, FFT, sorting, and reductions.
- GPU Kernel Differentiators - Computes gradients through GPU kernel code, enabling gradient-based optimization for simulation and machine learning.
- Tile-Based Kernel Differentiators - Generates backward-mode gradient computations for tile-based kernels, supporting in-place addition and subtraction.
- Array Converters - Provides zero-copy conversion of GPU arrays between Warp and JAX for seamless multi-framework workflows.
- JAX Array Interoperabilities - Converts Warp arrays to and from JAX arrays and creates JAX callbacks from Warp kernels.
- Kernel Primitives - Enables Warp GPU kernels to be called directly inside JAX JIT-compiled functions for accelerated computation.
- Finite Element Assemblers - Defines a geometry, function space, integration domain, and linear/bilinear forms to assemble and solve a system of equations.
- Physics Simulation - Runs large-scale physics simulations directly on GPU with automatic differentiation for gradient-based optimization.
- PyTorch Data Sharings - Converts arrays between Warp and PyTorch without copying data, preserving gradient information.
- GPU Device Synchronization - Synchronize kernel launches and memory copies automatically, requiring manual coordination only for asynchronous CPU transfers.
- Zero-Copy Array Slicing - Converts Warp arrays to and from any DLPack-compatible array without copying data.
- DLPack Protocols - Exchanges GPU arrays between frameworks via DLPack without copying memory for seamless interop.
- GPU Framework Data Exchanges - Exchanges GPU array data with PyTorch and JAX through the CUDA array interface without copying.
- GPU Kernel Function Wrappers - Wraps Python functions as Warp-compatible callables for use in GPU kernels.
- GPU-Accelerated Physics Simulations - Runs large-scale physics simulations such as particle systems with gravitational attraction directly on the GPU using custom kernel functions.
- GPU Stream Scheduling - Groups GPU operations into ordered sequences that execute concurrently on the same device to overlap compute and data transfers.
- Tile-Based Matrix Multiplications - Performs a general matrix multiply (GEMM) on tile operands, leveraging shared memory and Tensor Cores when available.
- Multi-Dimensional Arrays - Creates, clones, and manipulates fixed-size multi-dimensional arrays on CPU or CUDA devices.
- GPU Kernel Differentiators - Provides automatic differentiation through GPU kernel code for integration with PyTorch, JAX, and Paddle pipelines.
- GPU Orchestrators - Orchestrate array allocation, kernel launches, and data copies across multiple GPUs within a single process using device aliases.
- Compute Graph Captures - Records sequences of Warp and PyTorch operations into CUDA graphs for low-overhead repeated execution.
- Cross-Framework Graph Captures - Records a sequence of Warp and PyTorch kernel launches into a single CUDA graph for repeated replay.
- CUDA Graph Array Allocations - Records array creation as part of a CUDA graph capture so the graph allocates temporary storage when replayed.
- Multi-Device Kernel Launches - Launches independent kernels on different CUDA devices simultaneously to parallelize sub-tasks across GPUs.
- Kernel Caching Systems - Stores compiled GPU kernels between application runs to skip recompilation on subsequent launches.
- Multi-GPU Distribution - Uses JAX's shard_map to run Warp kernels on sharded arrays across multiple GPUs.
- Paddle Tensor Conversions - Converts Warp arrays to and from Paddle tensors without copying data, including gradient arrays.
- Tile Reductions - Computes the sum, minimum, maximum, or other aggregate of all elements in a tile cooperatively across a block.
- Tile Triangular System Solvers - Solves lower or upper triangular matrix equations on tile data using cooperative block-wide operations.
- Tile Reductions - Replaces per-thread atomic accumulation with a cooperative block-wide reduction to improve memory bandwidth utilization.
- JAX Array Conversions - Converts JAX arrays into Warp arrays using the DLPack protocol for zero-copy data exchange.
- JAX Callable Wrappers - Wraps Python functions that launch multiple Warp kernels as JAX callables for JIT-compiled code.
- Vectorized Mapping Behavior Controls - Specifies how Warp kernel callbacks transform under JAX's vmap transformation with broadcast or sequential options.
- Batched Environment Simulators - Represents topologically disconnected simulation environments within a single geometry for batched or reinforcement-learning workloads.
- GPU Kernel Runtime Checks - Activates runtime checks for array access, CUDA errors, floating-point values, and gradient computation to catch correctness issues during development.
- Mesh and BVH Queries - Finds closest points, intersections, and ray hits on meshes and bounding volume hierarchies.
- Tile-Based Spatial Queries - Performs axis-aligned bounding box or ray queries against a BVH or mesh within a tile, returning results cooperatively.
- GPU Array Atomics - Atomically reads, modifies, and writes values in GPU arrays for thread-safe concurrent updates.
- Compilation Strategy Selectors - Sets the compilation strategy for GPU kernels, choosing between eager, lazy, or other modes to balance startup time and runtime performance.
- GPU Math Type Constructors - Provides constructors for GPU math types like vectors, matrices, and quaternions used in physics and geometry kernels.
- Math Library Accelerators - Switches on GPU-accelerated implementations for FFT, matrix multiply, and solver operations using cuFFTDx, cuBLASDx, and cuSolverDx.
- Optimizer Integrations - Feeds Warp arrays to PyTorch optimizers like Adam by converting them to tensors for gradient updates.
- Spatial Field Evaluators - Looks up field values at arbitrary sample points using spatial queries and acceleration structures.
- Displacement Field Deformations - Deforms a base geometry with a displacement field to create a curved variant for higher-order finite element analysis.
- Field Couplings - Wraps a field defined on one geometry for integration over a different geometry using position-based mapping.
- 3D Matrix Builders - Builds, decomposes, and applies 4x4 transformation matrices from position, rotation, and scale components.
- Texture Sampling - Loads and samples 1D, 2D, and 3D textures with hardware acceleration on the GPU.
- Volumetric Mesh Extraction - Generates triangle meshes from volumetric data using a reusable marching cubes context on the GPU.
- Spatial Hierarchy Accelerators - Uses BVH and hash grids to accelerate collision detection and ray casting queries on the GPU.
- Remote GPU Memory Access - Permits one GPU to directly read or write memory allocated in another GPU's pool for accelerated cross-device transfers.
- Kernel Debugging Tools - Provides runtime checks, kernel code stepping, and CUDA Compute Sanitizer integration for debugging GPU kernels.
- Tile Pass-by-Reference Semantics - Allows tile parameters in user-defined functions to be modified in place, matching Python's mutable-object semantics.
- Adaptive Sparse Grids - Constructs a sparse grid with power-of-two voxel scales for adaptive resolution in finite element simulations.
- Finite Element Field Exporters - Generates VTK-compatible cell types and node indices from a function space for viewing in external tools.
- Sparse Matrix Multiplications - Performs GPU-accelerated matrix-matrix multiplication on BSR matrices with configurable scaling and accumulation.
- GPU Finite Element Solvers - Ships a finite element method solver that assembles and solves PDE systems entirely on GPU.
- Spatial Vector and Matrix Operations - Performs operations on 6D screw vectors and spatial inertia matrices for rigid-body dynamics.
- Matrix Transposition Kernels - Provides GPU-accelerated transposition of block-sparse matrices as part of its linear algebra engine.
- Block-Sparse Engines - Ships a block-sparse matrix engine for efficient GPU-accelerated linear algebra operations.
- NumPy Array Integration - Converts Warp arrays to and from NumPy arrays without copying data.
- Scalar Mathematical Utilities - Computes common mathematical functions such as trigonometry, exponentiation, and rounding on scalar values.
- Block-Sparse Matrix Arithmetic - Accumulates block-sparse matrices with scaling, reusing persistent GPU work buffers across calls.
- Sparse Matrix-Vector Multiplications - Computes a sparse matrix-vector product on the GPU, scaling and accumulating the result.
- Block-Sparse Matrix Builders - Builds block-sparse matrices from coordinate triplets with arbitrary block sizes on the GPU.
- Construction and Decomposition - Constructs, decomposes, and interpolates quaternions for representing 3D rotations in GPU kernels.
- GPU - Sorts key-value pairs in parallel using GPU-accelerated radix sort for efficient ordering.
- GPU Kernel Profilers - Records and reports the duration of individual CUDA operations such as kernel launches, memory copies, and memset calls.
