Open-source alternatives to ClipBERT

SlowFast is a PyTorch video understanding framework and spatiotemporal neural network library. It serves as a toolset for video action recognition, enabling the training and evaluation of models designed to classify complex activities and objects within video sequences. The framework is distinguished by its use of dual-pathway spatiotemporal sampling to capture both slow and fast motions. It supports self-supervised video learning for pre-training models on unlabeled data and employs multigrid spatiotemporal training to optimize learning across multiple spatial and temporal resolutions. The

Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It provides a pipeline for training large language models to process high-resolution images and video frames, integrating visual encoders with textual token spaces. The system is designed for multi-visual input processing, allowing models to interpret multiple images or video sequences within a single prompt. It supports multi-round conversation management to maintain context across interactions for detailed scene comprehension and visual reasoning. The framework covers a full develop

vjepa2 is a joint-embedding predictive architecture and video self-supervised learning framework. It functions as a visual representation learner and a robotic manipulation model designed to learn representations by predicting future latent states without reconstructing pixels. The system enables the pretraining of video encoders that learn temporally consistent features through masked-token prediction and multi-modal tokenization. It further maps these latent embeddings to specific physical movements via action-conditioned post-training to plan and execute robot arm grasping and picking task

mmaction2 is a PyTorch video understanding toolbox designed for training and evaluating deep learning models. It serves as a framework for action recognition, temporal localization, and spatio-temporal action detection, providing specialized tools for both pixel-based video analysis and skeleton-based action recognition. The project distinguishes itself through a modular architecture featuring registry-based component discovery and hierarchical, config-driven model assembly. It supports multi-modal feature fusion, integrating RGB frames, optical flow, and audio, and includes capabilities for

InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling

2022-06-24 (v0.3.4): Bugfixes.

Source code of our ACM MM'2022 paper Partially Relevant Video Retrieval.

source code of our paper Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning

Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".

Repository for Exposing the Limits of Video-Text Models through Contrast Sets (NAACL Short 2022).

Codes for ICCV2021 paper "COOKIE: Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation". [PDF](https://openaccess.thecvf.com/content/ICCV2021/html/WenCOOKIEContrastiveCross-ModalKnowledgeSharingPre-TrainingforVision-LanguageRepresentationICCV2021paper.html)

X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval Satya Krishna Gorti , Noël Vouitsis , Junwei Ma* , Keyvan Golestan , Maksims Volkovs , Animesh Garg , Guangwei Yu

Offical PyTorch implementation of Clover: Towards A Unified Video-Language Alignment and Fusion Model [arXiv](https://arxiv.org/abs/2207.07885)

W2VV++: A fully deep learning solution for ad-hoc video search. The code assumes video-level CNN features have been extracted.

Source code of our paper Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval.

A Joint Video and Image Encoder for End-to-End Retrieval project page | paper | dataset | demo Repository containing the code, models, data for end-to-end retrieval. WebVid data can be found here

Paper | Slide | Poster | Video

This repo contains code to evaluate for the semantic similarity video retrieval task, including: An example to generate a pandas dataframe from json annotations for YouCook2. A script to parse the captions using spacy. An optional script to create synset information using WordNet features. A…

This is the official implementation of the paper Frozen CLIP models are Efficient Video Learners

This is the official repository of VILLA (NeurIPS 2020 Spotlight). This repository currently supports adversarial finetuning of UNITER on VQA, VCR, NLVR2, and SNLI-VE. Adversarial pre-training with in-domain data will be available soon. Both VILLA-base and VILLA-large pre-trained checkpoints are…

PyTorch code for EMNLP 2020 Paper "Vokenization: Improving Language Understanding with Visual Supervision"

This repo provides code: - TeachText which leverages complementary cues from multiple text encoders to provide an enhanced supervisory signal to the retrieval model using a generalize distillation setup (paper, project page) - Learning and evaluating joint video-text embeddings for the task of…

This repo provides code from the HowTo100M paper. We provide implementation of: - Our training procedure on HowTo100M for learning a joint text-video embedding - Our evaluation code on MSR-VTT, YouCook2 and LSMDC for Text-to-Video retrieval - A pretrain model on HowTo100M - Feature extraction…

This github repo provides a Pytorch implementation of the Mixture-of-Embeddings-Experts model (MEE) 1.

News: The paper has been accepted to CVPR 2023!

This repo contains the code for the BMVC 2022 paper FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks.