hustvl/4DGaussians
4DGaussians
4DGaussians is a research library and neural rendering engine designed for reconstructing and rendering dynamic three-dimensional scenes. It represents moving environments as a collection of Gaussian primitives that evolve in position and appearance over a temporal dimension.
The framework utilizes neural deformation fields to predict spatial offsets and rotations for static point representations, simulating complex motion over time. It further employs temporal basis decomposition to encode motion trajectories into learned functions, compressing dynamic scene data while maintaining smooth transitions.
The system includes capabilities for real-time volumetric rendering, novel view synthesis, and high-resolution video generation. It provides a pipeline for dynamic mesh extraction, converting evolving point-based representations into three-dimensional mesh sequences, and supports multi-exposure synthesis for generating views with adjustable brightness and exposure.
The data pipeline handles camera pose processing and scene preprocessing using structure-from-motion, with utilities for exporting per-frame point cloud states for external analysis.
Features
- Gaussian Splatting - Provides a complete framework for training, rendering, and editing dynamic 3D scenes using Gaussian primitives.
- Multi-View Consistency Optimizers - Refines 3D representations by minimizing projection errors across synchronized multi-view video frames.
- Gaussian Splatting Optimizers - Optimizes scene geometry by backpropagating pixel-level reconstruction errors through a differentiable rasterization pipeline.
- Gaussian Splatting View Synthesizers - Generates novel viewpoints from 2D images by optimizing and rendering a 3D Gaussian representation.
- Dynamic Scene Training - Optimizes four-dimensional neural representations for temporally consistent reconstruction of moving scenes from video.
- Temporal Basis Decompositions - Encodes motion trajectories into learned basis functions to compress dynamic scene data and ensure smooth transitions.
- Deformation-Field Neural Mappings - Predicts spatial offsets and rotations for static point representations to simulate complex motion over time.
- Neural Rendering - Synthesizes novel views and 3D scenes using differentiable rendering of optimized Gaussian point clouds.
- Real-Time Neural Renderers - Synthesizes high-resolution video frames and novel camera views from neural representations with low latency.
- Neural Dynamic Rendering - Synthesizes high-resolution video frames in real-time by predicting point deformation across a timeline.
- Differentiable Rasterizers - Optimizes scene parameters by backpropagating pixel-level reconstruction errors through the projection and blending of geometric primitives.
- Gaussian Point Representations - Represents dynamic scenes as collections of anisotropic 3D ellipsoids that evolve over time.
- Computer Vision Research - Implements experimental neural rendering algorithms and measures reconstruction accuracy against ground truth for research purposes.
- 3D Asset Generators - Creates three-dimensional models from text or image inputs through iterative refinement of geometry and surface appearance.
- Dynamic View Synthesis - Generates images of reconstructed three-dimensional scenes from specified camera perspectives.
- Data Preprocessing Pipelines - Generates and downsamples point clouds from source images to establish a geometric foundation for training.
- Dynamic Mesh Reconstruction - Extracts high-quality three-dimensional mesh sequences from moving objects using two-dimensional representations and object masking.
- HDR Scene Renderers - Synthesizes novel camera views with adjustable brightness and exposure levels by learning from multi-exposure images.
- High-Resolution Rendering - Produces high-frame-rate visual output of dynamic scenes at professional resolutions.
- Gaussian-to-Mesh Converters - Converts evolving Gaussian point-based representations into high-quality three-dimensional mesh sequences.
- High-Fidelity Synthesis - Generates high-resolution image sequences that maintain structural integrity and realism for complex motion.
- Structure from Motion Pipelines - Generates initial point clouds and camera orientations from image sets using structure-from-motion pipelines.
- Tiled Rasterization - Accelerates visualization by projecting primitives into screen-space tiles for efficient hardware-accelerated alpha blending.
