Projects sharing features with MPLUG Owl

🌐 Homepage | 🤗 Paper | 📖 arXiv | 🤗 Dataset | 🏆 EvalAI | GitHub

Data and code for paper "M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models"

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

DeepSeek-VL2 is a multimodal large language model and vision-language system designed to analyze visual scenes and generate descriptive text. It functions as a visual question answering and visual grounding model, capable of extracting information from documents and locating specific objects or regions within images based on textual descriptions. The project utilizes a mixture-of-experts architecture to process combined image and text inputs. It is optimized for inference through incremental prefilling, which reduces the GPU memory requirements on hardware. The model covers multimodal data a

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

ICLR'24 Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

NAACL 2024 MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning

Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, Huaxiu Yao *Equal Contribution

LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

NeurIPS 2025 The official repository of "Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning"

VELOCITI Benchmark Evaluation and Visualisation Code

Official repository of MMGenBench

@Author: Qiguang Chen @LastEditors: Qiguang Chen @Date: 2024-05-23 20:24:16 @LastEditTime: 2024-05-26 18:09:00 @Description: -->

ACL 2024 Findings "TempCompass: Do Video LLMs Really Understand Videos?", Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou

NeurIPS 2024 MATH-Vision dataset and code to measure multimodal mathematical reasoning capabilities.

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Official implementation of "Visually Dehallucinative Instruction Generation: Know What You Don't Know"

VILA is a vision-language model integration that combines a visual encoder with a large language model to process images and text in a shared space. Its primary purpose is to enable the generation of natural language explanations and detailed text summaries of images and videos based on user prompts. The project utilizes a multi-stage alignment pipeline to synchronize visual and textual embeddings through sequential pretraining and supervised fine-tuning. To support deployment on desktop and edge hardware, it employs quantized low-precision inference to reduce model weights to 4-bit precision

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"

MiniCPM-o is a multimodal large language model designed to function as a real-time conversational assistant on edge devices. By mapping text, image, video, and audio inputs into a unified latent space, the system enables simultaneous cross-modal reasoning and full-duplex interaction. It is built as an edge-side inference engine, utilizing quantized model weights to maintain high-performance processing on consumer hardware. The system distinguishes itself through its integrated speech synthesis and voice cloning capabilities, which allow for the generation of expressive, personalized vocal out

CVPR2024 HighlightVideoChatGPT ChatGPT with video understanding! And many more supported LMs such as miniGPT4, StableLM, and MOSS.

InternVL is a vision-language model framework that fuses a visual encoder with a large language model to translate image features into textual tokens for reasoning. It provides a system for multimodal inference and dialogue, enabling the processing of images and text to answer questions or generate descriptions. The project is distinguished by its high-resolution image processing, which uses dynamic tiling to maintain detail for images up to 4K resolution, and its chain-of-thought visual reasoning for solving complex mathematical and spatial problems. It also supports temporal frame sampling

LLaMA-Adapter is a parameter-efficient fine-tuning framework designed to adapt large language models using a minimal set of trainable parameters. It functions as an instruction tuning tool and a multimodal adapter, allowing pre-trained models to follow human instructions and process non-textual data. The project specializes in the integration of image, video, audio, and sensor data into language models for cross-modal understanding. It enables the customization of LLaMA models through the use of lightweight adapters, which allows for the extraction and storage of learned weights independently