facebookresearch/sam-3d-body
Sam 3d Body
sam-3d-body is a machine learning framework for 3D human mesh recovery and pose estimation. It utilizes a 3D human mesh recovery model to reconstruct full-body meshes, including the body, hands, and feet, from a single image.
The project implements a specialized extension of the Segment Anything Model to guide the extraction and refinement of human body shapes. This integration allows for prompt-guided mesh recovery, where 2D masks and keypoints constrain the inference of 3D pose and shape parameters.
The system covers a range of computer vision capabilities, including 3D spatial alignment to place human meshes and objects into a shared coordinate frame and digital human modeling for virtual environments. It also includes tools for differentiable mesh rendering and visualization via reconstruction result overlays on original 2D images.
Features
- 3D Human Mesh Recovery - Reconstructs full-body 3D human meshes, including hands and feet, from a single input image.
- 3D Pose Estimation - Implements a framework for estimating human body pose and shape to align 3D mesh coordinates.
- Multi-Stage Inference Pipelines - Employs a multi-stage inference pipeline that chains segmentation, keypoint detection, and 3D mesh reconstruction.
- Parametric Human Model Fitting - Uses a statistical parametric model to decompose human body shape and pose into low-dimensional vectors.
- Pose Estimation - Estimates the three-dimensional position and shape of the human body using image data.
- Promptable 2D-to-3D Segmentations - Extends the Segment Anything Model to perform promptable 2D-to-3D segmentations for human body reconstruction.
- Mesh and Geometry Generators - Generates detailed 3D human body meshes using 2D keypoints and masks as guidance.
- Prompt-Guided Recovery - Allows auxiliary 2D keypoints or masks to guide the inference process for more accurate human mesh estimation.
- Human Digitalization - Creates detailed 3D representations of people from photographs for use in virtual environments.
- Silhouette and Keypoint Constrained Deformations - Constrains the inference of 3D pose and shape parameters using 2D masks and keypoint constraints.
- Image Segmentation - Integrates the Segment Anything Model to guide the precise extraction of human body shapes and meshes.
- Geometry-Based Mask Refinement - Iteratively adjusts 3D vertex positions to better align the mesh with provided 2D image mask boundaries.
- Differentiable Rendering - Provides differentiable mesh rendering to calculate loss based on the visual alignment of 3D vertices onto 2D image planes.
- Shared Coordinate Space Alignments - Implements a system to align human meshes and reconstructed objects into a single common 3D coordinate frame.
