Open-source alternatives to LAMM

✨✨CVPR 2025 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

Official implementation of "Visually Dehallucinative Instruction Generation: Know What You Don't Know"

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"

A VQA dataset that includes unanswerable questions TPAMI 2024.

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024)

Code for the paper Benchmarking Large Multimodal Models against Common Corruptions.

NeurIPS 2025 The official repository of "Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning"

ICLR'24 Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, Huaxiu Yao *Equal Contribution

VLMEvalKit is a vision-language model evaluation framework and inference engine designed to run standardized benchmarks and measure model accuracy across diverse visual datasets. It serves as a multimodal model benchmark and performance toolkit for calculating metrics and comparing model responses. The toolkit includes a specialized visual reasoning evaluator that uses adversarial samples to distinguish actual image understanding from reliance on language patterns. It also provides capabilities for image generation evaluation, testing a model's ability to create or modify visuals based on tex

This project provides a comprehensive Chinese language corpus designed to support the training and fine-tuning of large language models. It serves as a structured natural language processing resource, offering a collection of text data that includes dialogue, customer service interactions, and creative writing. The dataset is organized into distinct thematic categories, allowing for targeted model development across specific conversational and narrative contexts. By providing information in standardized, schema-agnostic text formats, the collection ensures portability across various machine l

This project is a technical curriculum and development guide focused on large language model prompt engineering, fine-tuning, and the creation of retrieval augmented generation applications. It serves as a comprehensive resource for developers to master crafting precise instructions and textual patterns to improve the quality and predictability of model outputs. The material covers the end-to-end workflow of adapting open-source models to specific datasets and integrating language models with vector databases to generate responses based on private information. It also provides a systematic ap

🤗 Dataset | 📖 arXiv | GitHub Atsuyuki Miyai 1   Jingkang Yang 2   Jingyang Zhang 3   Yifei Ming 4   Qing Yu 1,5   Go Irie 6   Sharon Yixuan Li 4   Hai Li 3   Ziwei Liu 2 Kiyoharu Aizawa 1 1 The University of Tokyo  2 S-Lab, Nanyang Technological…

This is the official repository of our paper ScanQA: 3D Question Answering for Spatial Scene Understanding (CVPR 2022) by Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoki Kawanabe. We propose a new 3D spatial understanding task for 3D question answering (3D-QA). In the 3D-QA task, models…

An automatic evaluation framework for Multimodal Chain-of-Thought.

🍎 Project Page 📖 arXiv Paper 📊 Dataset🏆 Leaderboard

This repo provides the source code & data of our paper: Evaluating Object Hallucination in Large Vision-Language Models (EMNLP 2023).

The jailbreak-evaluation is an easy-to-use Python package for language model jailbreak evaluation. The jailbreak-evaluation is designed for comprehensive and accurate evaluation of language model jailbreak attempts. Currently, jailbreak-evaluation support evaluating a language model jailbreak…

We introduce the task of dense captioning in 3D scans from commodity RGB-D sensors. As input, we assume a point cloud of a 3D scene; the expected output is the bounding boxes along with the descriptions for the underlying objects. To address the 3D object detection and description problems, we…

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

🌐 Homepage | 🤗 Paper | 📖 arXiv | 🤗 Dataset | 🏆 EvalAI | GitHub