Open-source alternatives to Cliport

This project is a computer control framework that uses multimodal vision models to simulate mouse and keyboard inputs for automating desktop tasks. It functions as an autonomous agent and vision-based orchestrator that interprets screen visuals to interact with user interfaces. The system employs vision language models and object detection to locate and click interface elements. It utilizes visual grounding to overlay numerical markers on UI components and uses optical character recognition to map on-screen text to precise pixel coordinates. The framework supports voice-controlled computing

Emu Series: Generative Multimodal Models from BAAI

Code for the paper "ViperGPT: Visual Inference via Python Execution for Reasoning"

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Official Repository for CVPR 2024 paper MultiPly: Reconstruction of Multiple People from Monocular Video in the Wild.

This repository contains the code for the paper What Can RL Bring to VLA Generalization? An Empirical Study. The pretrained checkpoints are available at HuggingFace.

Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming

Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting.

LLaVA-NeXT is a multimodal large language model framework and training toolkit designed to process interleaved images and video sequences to generate text. It functions as a visual language model that combines vision encoders with language models to perform complex reasoning, question answering, and video understanding. The system is capable of analyzing high-resolution images and temporal video frames to describe events, summarize actions, and reason across multiple visual inputs. It supports the interpretation of documents and charts, spatial environment analysis, and the generation of desc

Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration

MobileVLM: Vision Language Model for Mobile Devices

The ambition of the i-Code project is to build integrative and composable multimodal Artificial Intelligence. The "i" stands for integrative multimodal learning.

Official repo for MM-REACT

OmniParser is a multimodal interaction engine designed to function as a desktop automation agent. It interprets visual screen information to execute complex, multi-step tasks across operating system environments by bridging visual interface perception with language models. Through a continuous cycle of observation and command execution, the system grounds high-level natural language instructions into precise, coordinate-based actions. The project distinguishes itself by utilizing vision-based parsing to interact with software interfaces without requiring access to underlying application progr

Voyager is an autonomous embodied agent and lifelong learning framework that uses a large language model to explore virtual environments. It functions as a code-based action controller, translating natural language instructions into executable scripts to interact with its surroundings. The system features an automatic curriculum generator that creates sequences of exploration goals to discover new items and behaviors without human intervention. It maintains a skill library manager that stores learned behaviors as reusable code fragments, which can be composed to execute complex tasks. The fr

TMLR23 Official implementation of UnIVAL: Unified Model for Image, Video, Audio and Language Tasks.

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. (Correspondence )

[Paper](https://arxiv.org/abs/2210.05663) [Website](https://mahis.life/clip-fields/) [Code](https://github.com/notmahi/clip-fields) [Data](https://osf.io/famgv) [Video](https://youtu.be/bKu7GvRiSQU)

A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

MiniCPM-V is a multimodal large language model and vision-language system designed for complex visual and linguistic understanding. It functions as an on-device AI model, providing the capacity to process text, images, and video as a compact neural network. The project is specifically developed as an edge AI framework, utilizing quantization and weight sharding to run on memory-constrained mobile chipsets. This allows for the deployment of multimodal intelligence directly on mobile operating systems for local inference. Its capabilities cover multimodal content analysis of high-resolution im

AgiBot-World is a suite of software pipelines and tools designed for robotic policy training, dataset standardization, embodiment transfer, and performance benchmarking. It provides infrastructure for developing bimanual manipulation policies using foundation models and human-reference trajectory data. The project features a robot embodiment transfer suite that adapts pre-trained models to different robot bodies without requiring new multi-embodiment training data. It also includes a specialized evaluation framework for validating vision-language-action models through open-loop testing and ph