Open-source alternatives to VCoder

CVPR 2024 Prompt Highlighter: Interactive Control for Multi-Modal LLMs

ICLR'24 Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Pytorch implementation

LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

Efficient computing methods developed by Huawei Noah's Ark Lab

Official repo for StableLLAVA

Repo for LSKD: Distilling localized (e.g. bounding boxes), visual commonsense knowledge to Visual Language Models with ChatGPT generated data and filtering.

🧀 Code and models for the ICML 2023 paper "Grounding Language Models to Images for Multimodal Inputs and Outputs".

🐟 Code and models for the NeurIPS 2023 paper "Generating Images with Multimodal Language Models".

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. (Correspondence )

CLIP is a neural network architecture designed to map visual and textual data into a shared latent vector space. By utilizing transformer-based feature extraction and multi-modal tokenization, the system aligns images and natural language strings, enabling cross-modal similarity analysis and semantic classification. The project functions as a zero-shot classification engine, identifying image content by calculating the cosine similarity between visual features and arbitrary text labels without requiring task-specific retraining. Beyond inference, it serves as a research toolkit for evaluating

This project is a PyTorch implementation of a discrete variational autoencoder designed to compress high-resolution imagery into discrete latent representations. It functions as an image autoencoder that encodes visual data into discrete codes and decodes those codes back into reconstructed images. The system utilizes a latent space image compressor to convert images into a compressed, discrete format. This allows for generative image synthesis and the analysis of image compression by transitioning between raw pixels and discrete code sequences. The implementation covers latent space visuali

ECCV 2024 Best Paper Candidate & TPAMI 2025 PointLLM: Empowering Large Language Models to Understand Point Clouds

Bottom-up attention model for image captioning and VQA, based on Faster R-CNN and Visual Genome

Generative Adversarial Text-to-Image Synthesis

Understanding Complex Videos Relying on Large Language and Vision Models Project Page Paper The online demo is no longer available, because we released the code for offline demo deployment

Shikra : Unleashing Multimodal LLM’s Referential Dialogue Magic

TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models

We introduce a fine-grained visual instruction dataset, LVIS-INSTRUCT4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Please refer to the arxiv paper for more details.

ICML 2024 Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models.

Contextual Object Detection with Multimodal Large Language Models

NeurIPS-2023 Annual Conference on Neural Information Processing Systems