facebookresearch/sapiens
Sapiens
Sapiens is a high-resolution human vision model designed for high-precision, human-centric computer vision tasks. It functions as a suite of tools for estimating human pose, depth, and surface geometry.
The project utilizes a vision transformer backbone to perform multiple tasks through a shared encoder. This architecture enables the simultaneous prediction of skeletal structures, joint locations, and the distance between a camera and a human subject.
The model's capabilities cover human body part segmentation to isolate anatomical regions from backgrounds and surface normal prediction to recover 3D geometric details from 2D images. These tasks are supported by a multi-task learning framework that employs pixel-wise regression and semantic segmentation masking.
Features
- Human Pose Estimation - Predicts human anatomical keypoints and skeletal structures from 2D images.
- Human - Delineates human anatomical regions and joint locations to separate people from backgrounds.
- Depth Estimation - Calculates the distance between the camera and humans to create spatial scene representations.
- Human Body Part Segmentation - Provides high-resolution segmentation to isolate anatomical regions of the human body from backgrounds.
- Human-Centric Vision Systems - Performs high-precision vision tasks like pose and depth using a shared Vision Transformer backbone.
- Human Depth Estimation - Calculates the distance between a camera and a human subject to create spatial representations.
- Human Surface Normal Prediction - Determines the orientation of body surfaces to recover 3D geometric details from 2D images.
- Keypoint Detection - Implements anatomical landmark identification for human joints using heatmap-based probability distributions.
- Multi-Task Learning Models - Employs a shared encoder architecture to simultaneously perform depth, pose, and segmentation tasks.
- Shared-Bottom Architectures - Uses a shared-bottom architecture to extract features for multiple specialized human-centric output heads.
- Surface Normal Estimators - Determines the orientation of body surfaces to recover 3D geometric details from 2D images.
- Vision Transformers - Uses a transformer-based backbone to process images as patches for high-resolution feature extraction.
- Semantic Segmentation Architectures - Uses neural network structures to produce pixel-wise semantic labels for isolating human subjects.
- Human Centric Perception - Provides a high-resolution vision model for multi-modal human-centric tasks including pose, depth, and geometry.
- Surface Normal Estimation - Predicts the orientation of body surfaces to capture detailed structural information from images.
- Semantic Part Segmentation - Isolates specific anatomical regions using deep learning to partition the human subject.
- Human Body Part Segmenters - Identifies and delineates specific anatomical regions to separate human subjects from backgrounds.
- Feature Map Upsamplers - Utilizes decoder layers to increase feature map resolution for precise pixel-level predictions.
- Pixel-Wise Reconstruction Losses - Predicts continuous depth and surface normal values by minimizing pixel-wise reconstruction error.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
