facebookresearch/jepa
Jepa
This is a PyTorch self-supervised learning framework designed to train models that learn visual representations from video. It implements a joint-embedding predictive architecture that extracts spatio-temporal features by predicting missing regions of a signal within a latent representation space rather than reconstructing raw pixels.
The project includes a latent space visualization tool that uses a conditional diffusion model to decode feature-space predictions back into pixels. This allows for the verification of learned representations by transforming abstract predictions into interpretable images.
The framework provides a distributed training suite for executing large-scale pretraining and evaluation across multi-GPU clusters. It covers the end-to-end pipeline for video representation learning, including spatio-temporal data sampling, transformer-based feature extraction, and the evaluation of frozen encoders through linear probing and classification benchmarks.
Features
- Joint-Embedding Predictive Architectures - Implements a joint-embedding predictive architecture to learn representations by predicting latent states instead of pixels.
- Self-Supervised Learning - Implements a self-supervised learning framework that learns visual representations from unlabeled video data.
- Distributed Video Model Training - Enables large-scale pretraining and evaluation of video models across multi-GPU and distributed clusters.
- Large Scale Training Suites - Ships a comprehensive suite for orchestrating distributed pretraining and evaluation across multi-GPU clusters.
- Visual Representation Learning Frameworks - Provides a framework for training video encoders to capture complex spatiotemporal patterns and structures.
- Self-Supervised Vision Representation Trainers - Provides a PyTorch framework for learning visual representations from video using self-supervised predictive objectives.
- Transformer Feature Extractors - Uses a configurable transformer backbone to process spatio-temporal video data into high-dimensional latent vectors.
- Video Encoder Pretraining - Provides a pipeline for training transformer backbones to extract features from unlabeled video sequences.
- Self-Supervised - Learns visual features from unlabeled video data using a self-supervised predictive objective.
- Video Pre-training - Learns representations from unlabeled video data using self-supervised joint-embedding predictive objectives.
- Video Representation Learning - Extracts spatiotemporal features from video using a self-supervised objective that predicts missing latent regions.
- Data-Parallel Training - Provides a distributed training suite to synchronize model weights across multiple GPUs or clusters.
- Latent-to-Pixel Decoding - Converts feature-space predictions into pixels using a conditional diffusion model to visualize learned representations.
- Latent Decoding Visualizers - Includes a conditional diffusion model that decodes feature-space predictions into pixels for representation verification.
- Multi-Node Training Scaling - Executes large-scale pretraining and evaluation tasks across multi-GPU and distributed cluster configurations.
- Model Performance Evaluators - Quantifies the quality of learned representations by running classification benchmarks on video and image datasets.
- Representation Evaluation Tools - Assesses self-supervised visual features by running image classification tasks on frozen encoders.
- Spatio-Temporal Video Sampling - Prepares input data by applying spatial augmentations and temporal clips to capture motion and structure.
- Latent Visualization Decoders - Implements a conditional diffusion model to decode feature-space predictions back into pixels for representation verification.
- Linear Probing Evaluation - Evaluates representation quality by training a simple linear classifier on top of fixed model weights.
- Representation Quality Benchmarking - Tests the quality of learned visual representations by running frozen encoder weights on labeled video datasets.
- Generative Latent Visualizations - Implements a tool to verify learned representations by transforming abstract feature-space predictions into interpretable images.
- Video Dataset Processing - Loads video clips from index files and applies temporal sampling and spatial augmentations for model input.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
