Open-source alternatives to Seq Monkey Data

A VQA dataset that includes unanswerable questions TPAMI 2024.

Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

Official repo for StableLLAVA

NeurIPS 2025 The official repository of "Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning"

COYO-700M: Large-scale Image-Text Pair Dataset

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration

ACL 2024 🔥 Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.

Large Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.

Official implementation of "Visually Dehallucinative Instruction Generation" (ICASSP 2024)

Official implementation of "Visually Dehallucinative Instruction Generation: Know What You Don't Know"

ECCV2024 Video Foundation Models & Data for Multimodal Understanding

NeurIPS 2023 Datasets and Benchmarks Track LAMM: Multi-Modal Large Language Models and Applications as AI Agents

ECCV 2024 M3DBench introduces a comprehensive 3D instruction-following dataset with support for interleaved multi-modal prompts.

NeurIPS 2023 MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs

2023-06-13 Added tuned linear weights for Vicuna-7b. 2023-05-25 Our paper is available at this link. 2023-05-09 We have launched our project website. 2023-05-08 The first version of DetGPT is available now! Try our demo.

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

👾 E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024)

The official GitHub page for ''What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning''

GPT4Tools is an intelligent system that can automatically decide, control, and utilize different visual foundation models, allowing the user to interact with images during a conversation.

RedPajama-Data is a toolset for preprocessing large-scale text datasets used to train large language models. It provides a preprocessing pipeline focused on cleaning, deduplicating, and scoring massive collections of text to ensure data quality and diversity. The project utilizes a document quality scoring framework that employs machine learning and statistical heuristics to evaluate if documents are suitable for training. It includes a dataset filtering pipeline that uses classifiers and blocklists to remove undesirable words or URLs. The system features a text deduplication toolset that el

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

Project Page | Paper | Dataset