Awesome GitHub RepositoriesVision-Language Grounding Models

OmniParser is a multimodal interaction engine designed to function as a desktop automation agent. It interprets visual screen information to execute complex, multi-step tasks across operating system environments by bridging visual interface perception with language models. Through a continuous cycle of observation and command execution, the system grounds high-level natural language instructions into precise, coordinate-based actions. The project distinguishes itself by utilizing vision-based parsing to interact with software interfaces without requiring access to underlying application progr

Open-AutoGLM is an autonomous agent framework designed to perform complex user workflows on mobile devices. By translating natural language instructions into precise sequences of taps, scrolls, and text inputs, the system enables the automation of mobile application interactions and testing. The platform distinguishes itself through a combination of vision-language processing and reinforcement learning. It converts graphical user interfaces into structured data, allowing agents to parse screen elements and map natural language commands to coordinate-based actions. To ensure reliability, the s

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Grounded-Segment-Anything is a suite of specialized tools for multimodal visual analysis, text-based segmentation, and generative image editing. It integrates text-to-bounding-box detection and high-precision image segmentation masks to function as a text-based image segmenter and an automated visual labeling tool. The project enables text-driven image editing by identifying objects through natural language to perform inpainting and element replacement. It further extends visual analysis into three dimensions, allowing for 3D human reconstruction and the generation of 3D bounding boxes from t

Agent-S is a multimodal AI agent and LLM desktop automation framework designed to control operating systems through graphical user interface interactions. It functions as a computer use interface, utilizing vision-language grounding to translate natural language goals into precise screen coordinates and system actions. The project differentiates itself by combining structured accessibility tree inspection with vision-based element localization. It manages cross-application workflows by mapping conceptual descriptions to physical pixels and simulating low-level keyboard and mouse events to mov

Midscene is a multimodal automation framework designed to enable AI agents to perceive, navigate, and manipulate graphical user interfaces across web, mobile, and desktop environments. By leveraging vision-capable AI models, the platform interprets interface screenshots to execute tasks based on natural language instructions, removing the reliance on traditional, brittle code-based selectors. The framework distinguishes itself through its ability to decompose high-level goals into autonomous, multi-step sequences that function consistently across diverse platforms. It provides a visual ground

ml-ferret is a multimodal large language model framework and visual reasoning engine designed to reason about images and user interfaces. It functions as a UI grounding model and referring expression comprehension tool that maps natural language descriptions to precise pixel coordinates. The system focuses on high-resolution image analysis to identify and locate specific interface components. It employs multi-resolution image processing and region-aware visual encoding to preserve detail across different aspect ratios, enabling the model to analyze spatial relationships and functional layouts

Acest proiect oferă un framework fundamental și o implementare de referință pentru executarea modelării limbajului cauzal și a raționamentului multimodal pe sisteme locale. Include un set de componente de bază pentru gestionarea activelor modelului, un framework de fine-tuning și definițiile structurale necesare pentru a instanția arhitecturi bazate pe transformatoare. Sistemul se distinge prin capacitatea de a procesa intrări combinate de text și imagine prin modele transformatoare multimodale pentru raționament vizual și analiză de documente. De asemenea, suportă implementarea modelelor cuantizate, reducând amprenta de memorie prin tehnici de precizie scăzută pentru a permite inferența pe dispozitive edge. Proiectul acoperă domenii largi de capabilități, inclusiv fine-tuning supervizat și adaptare low-rank pentru personalizarea pe domeniu, precum și un manager de active cuprinzător pentru descărcarea, verificarea și organizarea ponderilor modelelor și a tokenizerelor. Funcționalitățile suplimentare includ generarea de text multilingv, procesarea contextului lung și ancorarea limbajului vizual (visual language grounding).

CogVLM is a multimodal large language model designed for visual reasoning and multi-turn dialogue. It functions as a visual grounding model and a quantized vision model, combining text and image processing to perform complex understanding and maintain context across visual inputs. The project includes capabilities as a GUI automation agent, allowing it to analyze application screenshots, plan operational steps, and return precise screen coordinates for interface interaction. It further supports visual grounding by generating bounding box coordinates to map text descriptions to specific spatia

DeepSeek-VL2 este un model de limbaj mare multimodal și un sistem vision-language conceput pentru a analiza scene vizuale și a genera text descriptiv. Funcționează ca un model de visual question answering și visual grounding, capabil să extragă informații din documente și să localizeze obiecte sau regiuni specifice în imagini pe baza descrierilor textuale. Proiectul utilizează o arhitectură mixture-of-experts pentru a procesa intrări combinate de imagine și text. Este optimizat pentru inferență prin prefilling incremental, ceea ce reduce cerințele de memorie GPU pe hardware. Modelul acoperă analiza datelor multimodale și înțelegerea documentelor vizuale, inclusiv interpretarea graficelor și a layout-urilor. Efectuează inferență vizuală și grounding pentru a potrivi interogările textuale cu conținutul vizual corespondent.