SysCV/sam-hq
Sam Hq
sam-hq is a collection of pre-trained vision foundation models and adapters designed for high-quality image segmentation, multimodal feature extraction, and depth estimation. It provides a zero-shot vision model capable of performing segmentation and classification across diverse domains without requiring task-specific training.
The project features a high-quality image segmentation tool based on the Segment Anything Model that generates precise masks from spatial prompts. It includes a multimodal feature extractor to generate high-dimensional vector embeddings from both image and text inputs, as well as a convolutional tool for predicting distance or canopy height from visual data.
The framework covers a broad range of computer vision capabilities, including image classification, multi-resolution feature extraction, and image preprocessing. It supports domain adaptation through fine-tuning on custom datasets for specialized applications such as medical imaging and remote sensing.
The mask decoder can be converted to an open format for execution in environments with a standard runtime.
Features
- Query-Based Mask Generators - Generates precise binary segmentation masks based on spatial points or bounding box queries.
- SAM-Based Implementations - Implements a high-quality image segmentation tool based on the Segment Anything Model architecture.
- High-Precision Mask Generators - Produces high-quality, precise binary segmentation masks for intricate objects using spatial prompts.
- Depth Estimation - Provides a convolutional head for predicting distance or canopy height from visual data.
- Multimodal Feature Extractors - Generates unified high-dimensional vector representations from both image and text inputs.
- Multi-Modal Embedding Models - Provides a multimodal embedding model that maps image and text data into a shared vector space.
- Vision Transformer Encoders - Uses vision transformer encoders to extract high-dimensional visual features from image patches.
- Zero-Shot Generalization - Enables the model to segment objects in unseen images without requiring task-specific training.
- Zero-Shot Segmentations - Creates fine-grained masks for objects based on user prompts without task-specific training.
- Zero-Shot Vision Foundation Models - Provides a pre-trained foundation model for zero-shot segmentation and classification across diverse domains.
- Depth Estimation - Predicts spatial distance or canopy height from visual data using a specialized depth estimation head.
- Transformer Embedding Extraction - Extracts raw hidden-state embeddings from transformer internals to serve as a foundation for vision tasks.
- High-Resolution Feature Extraction - Generates high-dimensional vector representations from images at various resolutions.
- Multi-Scale Feature Aggregation - Extracts image representations at multiple scales to capture both global context and local detail.
- Domain Adaptation - Supports fine-tuning segmentation models on custom datasets for medical imaging and remote sensing.
- Task-Specific Heads - Utilizes task-specific output heads to perform specialized functions like depth estimation and classification.
- Zero-Shot Image Classifiers - Categorizes images by comparing visual features against text labels without task-specific training.
- Domain-Specific Fine-Tuning - Provides a framework for fine-tuning segmentation masks for specialized applications like remote sensing.
- Vision Model Fine-Tuning - Supports adapting pre-trained vision models to specialized domains like medical imaging and remote sensing through fine-tuning.
