yangchris11/samurai
Samurai
SAMURAI is a zero-shot visual tracking model that adapts the Segment Anything architecture for video object segmentation. It uses a first-frame prompt, such as a bounding box or mask, to initialize tracking, then employs a motion-aware memory mechanism that stores and updates temporal motion features across frames to guide mask refinement. An online memory update strategy continuously refreshes this memory with new frame predictions, while temporal motion encoding computes optical flow between consecutive frames to inform object boundary and occlusion handling.
The system is designed for real-time inference, optimizing model forward passes and memory operations to achieve tracking speeds on standard hardware. It operates without requiring any task-specific fine-tuning or training, enabling zero-shot tracking of unseen objects in video. SAMURAI also handles objects that enter the video after the initial frame, detecting and following them once they appear.
The project provides a custom video demo runner that accepts a video file or frame directory and a first-frame bounding box to produce tracking results. It supports semi-supervised video object segmentation, generating segmentation predictions using a pre-trained model and ground-truth masks from the first frame.
Features
- Zero-Shot Video Tracking Pipelines - Adapts a pre-trained segmentation model to track unseen objects without task-specific fine-tuning.
- Motion-Guided Mask Refinements - Incorporates temporal motion cues to improve mask predictions for fast-moving or occluded objects.
- Segment Anything Model Adaptations - Repurposes the Segment Anything model for video tracking by injecting motion cues into its decoder.
- Motion-Aware Video Segmentors - Refines object segmentation masks by incorporating temporal motion cues for fast-moving or occluded objects.
- Semi-Supervised Video Object Segmentors - Generates segmentation masks for tracked objects using a first-frame annotation and a pre-trained model.
- Semi-Supervised Video Object Trackers - Generates segmentation predictions using ground-truth masks from the first frame and a pre-trained model.
- Semi-Supervised Video Segmentors - Generates segmentation predictions using a pre-trained model and ground-truth masks from the first frame.
- Zero-Shot Object Trackers - Leverages a motion-aware memory mechanism to track objects without requiring any training or fine-tuning.
- Zero-Shot Video Segmentors - Segments and tracks any object in video without prior training using motion-aware memory.
- Zero-Shot Video Trackers - Tracks any object in video without prior training or fine-tuning using a pre-trained model.
- First-Frame Prompt Initializations - Provides the initial bounding box or mask that seeds the entire tracking process.
- Optical Flow Encodings - Computes optical flow between consecutive frames to inform object boundary and occlusion handling.
- Temporal Motion Feature Memories - Stores and updates temporal motion features to guide mask refinement across frames.
- Real-Time Model Inference on Frames - Optimizes model forward passes and memory operations for real-time tracking on standard hardware.
- Online Memory Refreshes - Continuously refreshes motion-aware memory with new frame predictions for consistent tracking.
- Late-Appearing Object Trackers - Handles objects that enter the video after the initial frame by detecting and following them.
