This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal language models. It provides a comprehensive suite for vision-language pretraining, enabling the development of models that map images and text into shared latent spaces. The framework is distinguished by its capabilities in high-fidelity image generation and multimodal research, utilizing normalizing flows and variational autoencoders to produce images from text prompts or class labels. It supports the development of both generative and contrastive models, allowing for a wide
Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It provides a pipeline for training large language models to process high-resolution images and video frames, integrating visual encoders with textual token spaces. The system is designed for multi-visual input processing, allowing models to interpret multiple images or video sequences within a single prompt. It supports multi-round conversation management to maintain context across interactions for detailed scene comprehension and visual reasoning. The framework covers a full develop
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
This is the official PyTorch implementation of the ALBEF paper Blog . This repository supports pre-training on custom datasets, as well as finetuning on VQA, SNLI-VE, NLVR2, Image-Text Retrieval on MSCOCO and Flickr30k, and visual grounding on RefCOCO+. Pre-trained and finetuned checkpoints…
PyTorch code for EMNLP 2020 Paper "Vokenization: Improving Language Understanding with Visual Supervision"
الميزات الرئيسية لـ airsplay/vokenization هي: Multimodal Pretraining.
تشمل البدائل مفتوحة المصدر لـ airsplay/vokenization: google-research/big_vision — This project is a research framework and toolkit designed for training large-scale vision transformers and multimodal… evolvinglmms-lab/otter — Otter is a framework and toolkit for the pretraining, fine-tuning, and evaluation of vision-language models. It… jayleicn/clipbert — Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling. salesforce/albef — This is the official PyTorch implementation of the ALBEF paper [Blog] . This repository supports pre-training on… uclanlp/visualbert. airsplay/lxmert — Our servers break again :(. I have updated the links so that they should work fine now. Sorry for the inconvenience.…