facebookresearch/sam-3d-objects
Sam 3d Objects
SAM 3D Objects is a promptable foundation model that recovers 3D objects and human meshes from single images. It converts masked objects in a single photograph into full 3D models with pose, shape, texture, and layout, while also producing complete 3D human body meshes from the same input.
The system integrates promptable segmentation to isolate objects and humans before reconstruction, then aligns the independently reconstructed 3D elements into a shared coordinate space. This enables scene-level understanding where multiple 3D reconstructions from the same image coexist in a common coordinate frame.
The pipeline is end-to-end differentiable, combining segmentation, reconstruction, alignment, and texture recovery for joint optimization from monocular input. It uses learned parametric models and geometric reasoning to estimate 3D shape, pose, and texture from a single view.
Features
- 3D Reconstruction - Uses a promptable foundation model to recover 3D objects and humans from a single image.
- 3D Human Mesh Recovery - Produces a full 3D human mesh from a single image using a promptable foundation model.
- Prompt-Guided Reconstructions - Uses segmentation prompts to guide which objects in an image get reconstructed into 3D models.
- Single-Image 3D Reconstructions - Converts masked objects in a single photograph into full 3D models with pose, shape, texture, and layout.
- Multi-Object Scene Alignments - Aligns multiple 3D reconstructions of objects and humans from the same image into a shared coordinate space.
- Shared Coordinate Space Alignments - Aligns multiple 3D reconstructions from the same image into a shared coordinate space for scene-level understanding.
- Single-Image Scene Alignments - Combines 3D object and human reconstructions from the same image into a shared coordinate space.
- Single-View Shape and Texture Estimations - Estimates 3D shape, pose, and texture from a single image using learned parametric models.
- Differentiable 3D Reconstruction Pipelines - Combines segmentation, reconstruction, and alignment in a differentiable framework for joint optimization.
- Promptable 2D-to-3D Segmentations - Uses a promptable segmentation model to isolate objects and humans before 3D reconstruction.
- Mask-Guided Object Reconstructions - Reconstructs individual objects as complete 3D models with pose and shape from masked image regions.
- Textured Mesh Optimizations - Recovers surface texture and spatial layout from monocular input using differentiable rendering and optimization.
- Multi-Reconstruction Scene Alignments - Aligns independently reconstructed 3D objects and humans into a common coordinate frame using camera and scene geometry.
