Awesome GitHub RepositoriesVision-Language Grounding Models

OmniParser is a multimodal interaction engine designed to function as a desktop automation agent. It interprets visual screen information to execute complex, multi-step tasks across operating system environments by bridging visual interface perception with language models. Through a continuous cycle of observation and command execution, the system grounds high-level natural language instructions into precise, coordinate-based actions. The project distinguishes itself by utilizing vision-based parsing to interact with software interfaces without requiring access to underlying application progr

Open-AutoGLM is an autonomous agent framework designed to perform complex user workflows on mobile devices. By translating natural language instructions into precise sequences of taps, scrolls, and text inputs, the system enables the automation of mobile application interactions and testing. The platform distinguishes itself through a combination of vision-language processing and reinforcement learning. It converts graphical user interfaces into structured data, allowing agents to parse screen elements and map natural language commands to coordinate-based actions. To ensure reliability, the s

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Grounded-Segment-Anything is a suite of specialized tools for multimodal visual analysis, text-based segmentation, and generative image editing. It integrates text-to-bounding-box detection and high-precision image segmentation masks to function as a text-based image segmenter and an automated visual labeling tool. The project enables text-driven image editing by identifying objects through natural language to perform inpainting and element replacement. It further extends visual analysis into three dimensions, allowing for 3D human reconstruction and the generation of 3D bounding boxes from t

Agent-S is a multimodal AI agent and LLM desktop automation framework designed to control operating systems through graphical user interface interactions. It functions as a computer use interface, utilizing vision-language grounding to translate natural language goals into precise screen coordinates and system actions. The project differentiates itself by combining structured accessibility tree inspection with vision-based element localization. It manages cross-application workflows by mapping conceptual descriptions to physical pixels and simulating low-level keyboard and mouse events to mov

Midscene is a multimodal automation framework designed to enable AI agents to perceive, navigate, and manipulate graphical user interfaces across web, mobile, and desktop environments. By leveraging vision-capable AI models, the platform interprets interface screenshots to execute tasks based on natural language instructions, removing the reliance on traditional, brittle code-based selectors. The framework distinguishes itself through its ability to decompose high-level goals into autonomous, multi-step sequences that function consistently across diverse platforms. It provides a visual ground

ml-ferret is a multimodal large language model framework and visual reasoning engine designed to reason about images and user interfaces. It functions as a UI grounding model and referring expression comprehension tool that maps natural language descriptions to precise pixel coordinates. The system focuses on high-resolution image analysis to identify and locate specific interface components. It employs multi-resolution image processing and region-aware visual encoding to preserve detail across different aspect ratios, enabling the model to analyze spatial relationships and functional layouts

Dieses Projekt bietet ein grundlegendes Framework und eine Referenzimplementierung für die Ausführung von kausaler Sprachmodellierung und multimodalem Reasoning auf lokalen Systemen. Es enthält eine Reihe von Kernkomponenten für die Verwaltung von Modell-Assets, ein Fine-Tuning-Framework sowie strukturelle Definitionen, die für die Instanziierung von Transformer-basierten Architekturen erforderlich sind. Das System zeichnet sich durch die Fähigkeit aus, kombinierte Text- und Bildeingaben durch multimodale Transformer-Modelle für visuelles Reasoning und Dokumentenanalyse zu verarbeiten. Es unterstützt zudem die Bereitstellung quantisierter Modelle, wodurch der Speicherbedarf durch Techniken mit niedriger Präzision reduziert wird, um Inferenz auf Edge-Geräten zu ermöglichen. Das Projekt deckt breite Funktionsbereiche ab, einschließlich Supervised Fine-Tuning und Low-Rank Adaptation für die Domänenanpassung sowie einen umfassenden Asset-Manager zum Herunterladen, Verifizieren und Organisieren von Modellgewichten und Tokenizern. Zusätzliche Funktionen umfassen mehrsprachige Textgenerierung, Verarbeitung langer Kontexte und Visual Language Grounding.

CogVLM is a multimodal large language model designed for visual reasoning and multi-turn dialogue. It functions as a visual grounding model and a quantized vision model, combining text and image processing to perform complex understanding and maintain context across visual inputs. The project includes capabilities as a GUI automation agent, allowing it to analyze application screenshots, plan operational steps, and return precise screen coordinates for interface interaction. It further supports visual grounding by generating bounding box coordinates to map text descriptions to specific spatia

DeepSeek-VL2 ist ein multimodales Large Language Model und Vision-Language-System, das darauf ausgelegt ist, visuelle Szenen zu analysieren und beschreibenden Text zu generieren. Es fungiert als Modell für visuelle Fragenbeantwortung (VQA) und visuelle Verankerung (Visual Grounding), das in der Lage ist, Informationen aus Dokumenten zu extrahieren und spezifische Objekte oder Regionen innerhalb von Bildern basierend auf textuellen Beschreibungen zu lokalisieren. Das Projekt nutzt eine Mixture-of-Experts-Architektur, um kombinierte Bild- und Texteingaben zu verarbeiten. Es ist für die Inferenz durch inkrementelles Prefilling optimiert, was den GPU-Speicherbedarf auf Hardware reduziert. Das Modell deckt multimodale Datenanalyse und visuelles Dokumentenverständnis ab, einschließlich der Interpretation von Diagrammen und Layouts. Es führt visuelle Inferenz und Verankerung durch, um textuelle Anfragen mit entsprechenden visuellen Inhalten abzugleichen.