Open-source alternatives to LVIS INSTRUCT4V

Pytorch implementation

LLaVA is a multimodal large language model architecture designed to process and interpret both image and text inputs to generate natural language responses. It functions as a research-oriented platform for visual instruction tuning, providing a framework to align language models with human intent through training on diverse datasets of paired images and text queries. The system distinguishes itself through a specialized vision-language training pipeline that connects visual data to language models using projection layers and instruction-based fine-tuning. It supports distributed inference by

Official repo for StableLLAVA

Repo for LSKD: Distilling localized (e.g. bounding boxes), visual commonsense knowledge to Visual Language Models with ChatGPT generated data and filtering.

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. (Correspondence )

CLIP is a neural network architecture designed to map visual and textual data into a shared latent vector space. By utilizing transformer-based feature extraction and multi-modal tokenization, the system aligns images and natural language strings, enabling cross-modal similarity analysis and semantic classification. The project functions as a zero-shot classification engine, identifying image content by calculating the cosine similarity between visual features and arbitrary text labels without requiring task-specific retraining. Beyond inference, it serves as a research toolkit for evaluating

This project is a PyTorch implementation of a discrete variational autoencoder designed to compress high-resolution imagery into discrete latent representations. It functions as an image autoencoder that encodes visual data into discrete codes and decodes those codes back into reconstructed images. The system utilizes a latent space image compressor to convert images into a compressed, discrete format. This allows for generative image synthesis and the analysis of image compression by transitioning between raw pixels and discrete code sequences. The implementation covers latent space visuali

ECCV 2024 Best Paper Candidate & TPAMI 2025 PointLLM: Empowering Large Language Models to Understand Point Clouds

Bottom-up attention model for image captioning and VQA, based on Faster R-CNN and Visual Genome

Generative Adversarial Text-to-Image Synthesis

Understanding Complex Videos Relying on Large Language and Vision Models Project Page Paper The online demo is no longer available, because we released the code for offline demo deployment

CVPR 2024 VCoder: Versatile Vision Encoders for Multimodal Large Language Models

Shikra : Unleashing Multimodal LLM’s Referential Dialogue Magic