facebookresearch/co-tracker

Features

• Dense Pixel Tracking - Provides dense pixel tracking capabilities to map the motion of individual pixels throughout a video.
• Differentiable Visibility Masks - Predicts binary visibility scores for each tracked point to identify when pixels leave the frame or are occluded.
• Point Tracking - Tracks individual or dense sets of pixels across video frames while handling occlusions and object exits.
• Dense Tracking Architectures - Implements a neural network architecture that maps the motion of individual pixels throughout a video.
• Teacher-Student Distillation - Uses teacher-student knowledge distillation to train models on unannotated real video data via pseudo-labels.
• Point Tracking Frameworks - Provides a PyTorch-based framework for tracking dense sets of pixels across video frames using deep learning.
• Model-Assisted Labelers - Generates labels from teacher models using unannotated real videos to bridge the gap between synthetic and real data.
• Visual Point Trackers - Provides a system for predicting point trajectories and visibility masks across sequences of video frames.
• Pseudo-Label Video Generation - Generates ground truth pseudo-labels for unannotated videos using teacher models to train more robust trackers.
• Pixel Motion Analysis - Analyzes point trajectories and visibility over time to understand pixel movement within video sequences.
• Iterative Refinement Workflows - Utilizes iterative refinement loops to update point locations and correct drift over long video sequences.
• Pixel-Level Attention - Implements pixel-level cross-attention mechanisms to compute correspondences between feature vectors across video frames.
• Video Sequence Architectures - Processes video frames in both forward and backward directions to maintain tracking continuity during occlusions.
• Computer Vision Training - Provides a training pipeline for fine-tuning point tracking models using teacher-student distillation.
• Model Fine-Tuning - Allows fine-tuning of pre-trained tracking models on real video datasets to improve performance in specific environments.
• Pseudo-Label Fine-Tuning - Updates pre-trained models using real video datasets and automatically generated labels to improve performance.
• Model Performance Evaluators - Measures the precision of predicted pixel trajectories against ground truth data to validate performance.
• Tracking Visualization - Renders predicted point trajectories and visibility masks over original video frames to inspect accuracy.
• Visual Tracking Evaluation - Provides utilities to measure tracking performance by comparing predicted pixel trajectories against ground truth data.
• Video Input Processing - Supports both offline context and online streaming processing for video sequence analysis.
• Frame Memory Buffers - Maintains a sliding-window buffer of previous frames to optimize memory and computational overhead during online inference.

Star history