facebookresearch/sam-3d-body

Features

• 3D Human Mesh Recovery - Reconstructs full-body 3D human meshes, including hands and feet, from a single input image.
• 3D Pose Estimation - Implements a framework for estimating human body pose and shape to align 3D mesh coordinates.
• Multi-Stage Inference Pipelines - Employs a multi-stage inference pipeline that chains segmentation, keypoint detection, and 3D mesh reconstruction.
• Parametric Human Model Fitting - Uses a statistical parametric model to decompose human body shape and pose into low-dimensional vectors.
• Pose Estimation - Estimates the three-dimensional position and shape of the human body using image data.
• Promptable 2D-to-3D Segmentations - Extends the Segment Anything Model to perform promptable 2D-to-3D segmentations for human body reconstruction.
• Mesh and Geometry Generators - Generates detailed 3D human body meshes using 2D keypoints and masks as guidance.
• Prompt-Guided Recovery - Allows auxiliary 2D keypoints or masks to guide the inference process for more accurate human mesh estimation.
• Human Digitalization - Creates detailed 3D representations of people from photographs for use in virtual environments.
• Silhouette and Keypoint Constrained Deformations - Constrains the inference of 3D pose and shape parameters using 2D masks and keypoint constraints.
• Image Segmentation - Integrates the Segment Anything Model to guide the precise extraction of human body shapes and meshes.
• Geometry-Based Mask Refinement - Iteratively adjusts 3D vertex positions to better align the mesh with provided 2D image mask boundaries.
• Differentiable Rendering - Provides differentiable mesh rendering to calculate loss based on the visual alignment of 3D vertices onto 2D image planes.
• Shared Coordinate Space Alignments - Implements a system to align human meshes and reconstructed objects into a single common 3D coordinate frame.

Star history