Mmdetection3d

MMDetection3D is an open-source toolbox for 3D perception, providing a unified framework for detecting and segmenting objects in three-dimensional environments. It supports a range of core tasks including monocular 3D object detection from single camera images, LiDAR-based 3D object detection from raw point clouds, and multi-modal fusion that combines camera images with LiDAR data. The toolbox also covers point cloud semantic segmentation, assigning class labels to every point in a scan for scene understanding.

The project distinguishes itself through a config-driven pipeline that orchestrates the entire training, evaluation, and inference workflow, with support for distributed training across multiple GPUs and machines. It includes a registry-based module composition system for assembling custom models from encoder, backbone, neck, head, and loss components, and provides built-in support for sparse convolution acceleration using libraries like spconv and MinkowskiEngine. The toolbox also offers a unified dataset format conversion system that transforms raw sensor data from benchmarks such as KITTI, Waymo, and nuScenes into a standardized internal structure, along with checkpoint-based training resumption and mixed precision training for fault-tolerant and efficient workflows.

Beyond its core detection and segmentation capabilities, the project provides a comprehensive set of tools for data preparation, augmentation, and evaluation. It includes data structuring for LiDAR, multi-modal, and vision-based detection tasks, point cloud augmentation techniques, and dataset-specific evaluation protocols with metrics like mean Average Precision. The toolbox also supports model deployment, leaderboard submission for autonomous driving benchmarks, and integration with over 500 pre-trained 2D detection models from a shared codebase. Installation is available via pip or the MIM tool, and the project can be run in Docker containers or on Windows for cross-platform compatibility.

Features

Custom Registrations - Register a new backbone network and integrate it into the model pipeline for extracting feature maps.
Bounding Box Representations - Stores and manipulates 3D bounding boxes with position, dimensions, yaw rotation, and coordinate system conversions.
Detection Head Registrations - Register a new bbox or RoI head and integrate it into the model pipeline for predicting object locations and classes.
Training Execution - Starts training jobs for 3D detection models using prepared configurations and datasets.
3D Point Cloud Custom Training - Converts raw point cloud data from custom sensors into structured formats for training 3D detection models.
Sensor Data Preparations - Converts raw sensor data into structured annotation and database files for model training.
3D Detection Pipeline Configurations - Enables adapting 3D detection pipelines to new datasets through configuration files.
Detection Loss Registrations - Register a new loss function and apply it to a specific head's loss field for training the model.
3D Detection Model Zoos - Run inference with pre-trained 3D detection models from a model zoo covering lidar, camera, and multi-modal architectures.
3D Detection Metrics - Computes standard 3D detection metrics like mAP and NDS on benchmark datasets during or after training.
Model Component Registries - Assemble a 3D detection model by plugging in custom encoder, backbone, neck, head, RoI extractor, or loss modules through a registry.
Model Evaluation Metrics - Run a trained model on the KITTI test set and compute mean Average Precision and Average Orientation Similarity metrics.
3D Detection Evaluations - Run trained 3D detection models on test data and compute metrics using dataset-specific evaluation protocols.
3D Detection Model Configurations - Provides config-driven adjustment of 3D detection model parameters for custom datasets.
Structured Experiment Configurations - Provides a structured config system for managing training, testing, and model parameters.
Point Cloud Segmentation Executions - Runs a pre-trained segmentation model on a point cloud and saves or displays the predicted per-point class labels.
LiDAR-Camera Fusions - Implements LiDAR-camera data fusion pipelines that synchronize point clouds and images for 3D detection.
LiDAR Detection Data Structuring - Organizes raw point cloud files and annotations into a prescribed folder layout for LiDAR detection.
3D Object Detection - Trains and evaluates a variety of 3D detection architectures on point cloud and multi-modal datasets.
Monocular Detections - Provides monocular 3D object detection from single camera images with known calibration.
Multi-Modal Detections - Ships multi-modal 3D detection that fuses camera images with LiDAR point clouds for improved accuracy.
Point Cloud Detections - Trains and evaluates 3D object detectors on raw LiDAR point clouds to predict oriented bounding boxes.
Inference Executions - Runs trained 3D detection models on new point cloud or multi-modal data to produce object predictions.
Training Executions - Trains 3D object detection models on point cloud or multi-modal data using a provided baseline configuration.
Training Pipelines - Loads point clouds from multiple sweeps and applies augmentations to train 3D object detectors on LiDAR data.
LiDAR-Based Training - Loads point clouds from multiple sweeps and applies augmentations like rotation and flipping for 3D detection.
Point Cloud and 3D Processing - Provides 3D semantic segmentation of point clouds for scene understanding tasks like road marking and indoor object segmentation.
3D Point Cloud Representations - Stores and manipulates 3D point cloud data with spatial coordinates and optional attributes like color and height.
LiDAR Semantic Segmentation Data Structuring - Organizes point cloud files and semantic masks into a prescribed layout for LiDAR segmentation.
Semantic Segmentations - Assigns semantic class labels to every point in LiDAR scans and indoor point clouds for scene understanding.
Augmentations - Applies random noise, flipping, rotation, and scaling to point cloud data during training.
Vision Detection Data Structuring - Organizes camera images and calibration files into a prescribed layout for vision-based detection.
Model Evaluation - Evaluates trained 3D detection models against standard benchmarks on datasets like KITTI, nuScenes, and Waymo.
3D Detection Benchmarks - Benchmarks trained models against standard indoor and outdoor 3D detection datasets with built-in evaluation pipelines.
Training Data Pipelines - Assembles sequences of operations to load, transform, and format point cloud data for model training.
Sensor Synchronizations - Loads and synchronizes point clouds, camera images, and calibration data from multiple sweeps into a unified input.
Dataset Preparation Scripts - Downloads, organizes, and preprocesses supported 3D datasets into expected folder structures and annotation formats.
Multi-Modal Detection Data Structuring - Organizes point clouds, images, and calibration files into a prescribed layout for multi-modal detection.
Sensor Dataset Conversions - Transforms raw sensor data from KITTI, Waymo, and nuScenes into a standardized internal structure for model consumption.
Inference Executions - Runs pre-trained 3D detection and segmentation models on new sensor data to generate predictions.
KITTI Format Converters - Converts raw point cloud data into KITTI-format annotations and info files for training.
Voxel Encoder Registrations - Register a new voxel or middle encoder and integrate it into the model pipeline for processing point cloud data.
Configuration-Driven Pipelines - Controls the entire training, evaluation, and inference workflow through hierarchical configuration files.
Custom Dataset Definitions - Adapts the data pipeline to support custom datasets for training or evaluating 3D detection models.
Dataset Class Balancing - Uses a class-balanced group sampling wrapper to handle imbalanced class distributions.
Dataset Registration Systems - Provides a mechanism for registering custom dataset classes to load annotation and label data.
Detection Visualization - Renders predicted 3D bounding boxes on images for qualitative inspection of model outputs.
Large-Scale Model Training - Distributes training across several machines or GPUs using Slurm or direct ethernet connections.
Learning Rate Decay Schedules - Supports configuring learning rate decay schedules like step, cosine annealing, and polynomial decay.
Mixed Precision Training - Accelerates training and reduces memory usage by performing operations in half-precision floating point.
Distributed Executions - Scales model training across multiple GPUs and machines with optional half-precision floating point for faster throughput.
Waymo Detection Metrics - Computes standard Waymo detection metrics by building and running the official evaluation binary.
Training Runtime Hook Configurations - Configures built-in hooks for logging, checkpoint saving, and visualization during training.
2D Model Integrations - Integrates over 500 pre-trained 2D detection models from a shared codebase alongside 3D detection workflows.
Model Performance Evaluators - Computes mean intersection over union across semantic classes to measure model performance.
nuScenes Leaderboard Submissions - Produce prediction files in the required format for submission to the nuScenes benchmark leaderboard.
Vision Model Training - Trains a monocular 3D object detection model using distributed GPU training with automatic learning rate scaling.
Bird's-Eye-View Training Pipelines - Loads multi-view images and applies augmentations to train bird's-eye-view 3D object detectors.
Gradient Clipping Configurations - Provides a unified wrapper for configuring optimizers, learning rates, and gradient clipping.
Pre-trained Model Application - Uses a library of over 500 pre-trained models to perform inference on new point cloud or multi-modal data.
Autonomous Driving Submissions - Formats model predictions into leaderboard-compatible submission files for KITTI, Waymo, and nuScenes.
Test Time Augmentation - Applies double flip augmentation during test time for improved CenterPoint detection accuracy.
Training and Testing Splits - Selects which S3DIS areas to use for training and validation by specifying area indices.
Checkpoint Resumption - Automatically resumes interrupted training from the latest saved checkpoint, preserving optimizer states and iteration progress.
Custom Point Cloud Dataset Training - Provides a pipeline for training 3D detection models on custom point cloud datasets with user-provided annotations.
Waymo 3D Perception Dataset Processors - Preprocesses, trains, tests, and evaluates models on the Waymo dataset with accelerated data handling.
Training Loop Control - Supports choosing between epoch-based and iteration-based training loops with dynamic validation intervals.
Segmentation Evaluation Metrics - Ships evaluation pipelines that compute per-class and mean IoU for 3D segmentation models.
Training Pipelines - Loads single-view images with camera intrinsics and 3D annotations to train monocular 3D object detectors.
Multi-Camera Detections - Supports transformation for multi-camera 3D object detection.
Sparse 3D Convolutional Networks - Integrates the MinkowskiEngine library for sparse 3D convolutional neural networks on point clouds.
Point-Cloud-Only Training - Supports training 3D detection models on datasets containing only point cloud data without image modalities.
3D Detection Benchmarks - Provides baseline results for numerous 3D detection models across standard datasets.
Format Converters - Converts point cloud files from PCD and LAS formats into BIN format for training.
3D Sensor Data Format Converters - Transforms raw point cloud and image files from vendor formats into KITTI-style representations for training.
Coordinate System Mapping - Transforms point coordinates, box dimensions, and yaw angles between camera, LiDAR, and depth coordinate systems.
Data Transformation Registrations - Ships a registry for adding custom data transformation steps into processing pipelines.
Dataset Configuration Systems - Sets up the data pipeline, dataloader, and evaluator in a config file to train and validate on a custom dataset.
KITTI 3D Perception Dataset Preparers - Converts raw KITTI point cloud and annotation files into structured formats for 3D detection training.
Format Conversions - Converts raw point cloud and annotation data into standard formats like KITTI for training pipeline compatibility.
Model Inference Deployment - Converts trained 3D detection models into deployable formats and serves them for production inference.
Benchmark Submissions - Formats and packages model predictions for upload to autonomous driving dataset leaderboards.
Sparse Voxel Operations - Accelerates 3D point cloud processing using sparse convolutional libraries like spconv and MinkowskiEngine.
Point Cloud Filtering - Filters background points from point cloud data during preprocessing to reduce noise.
Sparse Convolution Libraries - Integrates the spconv 2.0 library for accelerated sparse convolution on point cloud data.
Module-Based Registries - Assembles 3D detection models by registering and composing custom encoder, backbone, neck, head, and loss modules.
Training Lifecycle Hooks - Provides hooks for injecting custom logic at specific stages of the training lifecycle.
Autonomous Driving Benchmark Submissions - Generates prediction files and submits results to KITTI, nuScenes, and Waymo leaderboards for standardized comparison.
Computer Vision - Platform for 3D object detection.
Point Cloud Processing - Platform for general 3D object detection.

open-mmlab/OpenPCDet

5,621View on GitHub

OpenPCDet is a PyTorch deep learning library and toolbox for LiDAR 3D object detection. It functions as a point cloud processing framework designed to develop, train, and evaluate machine learning models that identify and locate objects in three dimensional space. The project includes a GPU-accelerated geometry engine for high-performance implementation of 3D intersection over union and rotated non-maximum suppression. It also provides a distributed model training tool to scale the training and testing of detection models across multiple GPUs and computing nodes. The framework covers point c

charlesq34/pointnet

5,433View on GitHub

PointNet is a deep learning architecture designed to process and classify raw 3D point clouds directly without voxelization. It provides a system for 3D object classification, semantic segmentation frameworks for partitioning clouds into categories, and tools for visualizing 3D shapes. The project utilizes a transform network to align point clouds into a canonical coordinate space and employs symmetric-function-based aggregation to condense point-wise features into global vectors regardless of point order. It also features a multi-scale grouping architecture to extract hierarchical geometric

CloudCompare/CloudCompare

4,577View on GitHub

CloudCompare is a professional software application for processing and analyzing 3D point clouds and polygonal meshes. It functions as a 3D mesh analysis tool and a large dataset visualizer designed to display and manage millions of points in a 3D environment. The software provides specialized capabilities for point cloud comparison, utilizing an optimized octree structure to calculate spatial differences between two 3D datasets. This allows for the identification of variations and errors between point clouds or between a point cloud and a mesh. The system covers broad 3D data analysis areas

udacity/self-driving-car

6,312View on GitHub

This is an open-source autonomous driving perception pipeline that processes camera and lidar sensor data to detect, track, and fuse objects in real-world driving environments. The project integrates an end-to-end perception workflow combining sensor calibration, deep learning object detection, Kalman filter tracking, and sensor fusion for robust scene understanding. The pipeline includes camera calibration tools to remove lens distortion from raw images, deep learning model training for object classification and detection, and multi-object tracking using Kalman filters with data association

open-mmlabmmdetection3d

Features