nianticlabs/monodepth2
Monodepth2
This project is a computer vision system for monocular depth estimation and 3D point cloud generation. It provides a supervised depth learning framework and a depth predictor capable of estimating spatial distance and disparity from single 2D images using pretrained neural networks.
The system includes tools to transform 2D depth images into 3D point clouds via pixel coordinate backprojection and converts 3D point cloud data into 2D depth maps. It utilizes a training pipeline that supports model fine-tuning and hyperparameter optimization.
The library covers broader capabilities in spatial analysis, including the calculation of rotation and translation matrices, depth map evaluation against ground truth benchmarks, and pose prediction assessment. It also provides utilities for pixel normalization and checksum-verified retrieval of pretrained model weights.
Features
- Monocular Depth Estimators - Provides monocular depth estimation to predict scaled disparity or metric depth maps from single 2D images.
- Depth Estimation Training - Provides a full framework for training neural networks to estimate relative or absolute depth from image datasets.
- Depth Map Evaluation - Implements utilities to measure the accuracy and precision of predicted depth maps against ground truth benchmarks.
- Convolutional Encoder-Decoders - Utilizes a convolutional encoder-decoder architecture to extract image features and upscale them into high-resolution depth maps.
- Depth Estimation Fine-Tunings - Enables the refinement of pretrained depth models on specific target datasets for improved spatial accuracy.
- Vision Model Training - Ships a training pipeline for optimizing neural networks to recognize spatial relationships and distance from image datasets.
- Supervised Learning Frameworks - Provides a supervised learning framework for training and evaluating networks to predict image depth.
- 3D Point-Cloud Spatial Mappers - Generates 3D coordinate sets from 2D images and estimated depth data for spatial analysis.
- Image-to-Point Cloud Transformers - Transforms 2D depth maps into 3D point cloud representations using camera intrinsic parameters.
- Relative Depth Estimators - Predicts relative depth ordering and disparity to handle varying camera focal lengths and scene sizes.
- Hyperparameter Optimization - Supports automated tuning of learning rates, batch sizes, and epoch counts to improve model convergence.
- Model Prediction Evaluation - Includes tools to assess the accuracy of predicted spatial poses by comparing them against ground truth odometry data.
- Self-Supervised Loss Calculators - Uses a self-supervised photometric loss to train models by comparing predicted depth against neighboring video frames.
- Spatial Vector Calculators - Calculates 4x4 transformation matrices from axis-angle rotations and translation vectors for 3D spatial mapping.
- Camera Pose Estimators - Implements compact axis-angle representations to calculate camera orientations and positions from video frames.
- Spatial Mapping - Creates digital 3D representations by calculating rotation and translation matrices from 2D images.
- Depth estimation - Listed in the “Depth estimation” section of the Ailia Models awesome list.
