Awesome GitHub RepositoriesVision-Language Grounding Models

OmniParser is a multimodal interaction engine designed to function as a desktop automation agent. It interprets visual screen information to execute complex, multi-step tasks across operating system environments by bridging visual interface perception with language models. Through a continuous cycle of observation and command execution, the system grounds high-level natural language instructions into precise, coordinate-based actions. The project distinguishes itself by utilizing vision-based parsing to interact with software interfaces without requiring access to underlying application progr

Open-AutoGLM is an autonomous agent framework designed to perform complex user workflows on mobile devices. By translating natural language instructions into precise sequences of taps, scrolls, and text inputs, the system enables the automation of mobile application interactions and testing. The platform distinguishes itself through a combination of vision-language processing and reinforcement learning. It converts graphical user interfaces into structured data, allowing agents to parse screen elements and map natural language commands to coordinate-based actions. To ensure reliability, the s

This project is a comprehensive framework and toolkit for developing, optimizing, and deploying transformer-based models across multimodal, document intelligence, and natural language processing tasks. It provides a unified neural architecture that processes text, vision, audio, and document layout data through a shared set of weights, enabling researchers and developers to build foundational models that align cross-modal representations. The platform distinguishes itself through advanced training and inference strategies designed for large-scale deep learning. It incorporates specialized mec

Grounded-Segment-Anything is a suite of specialized tools for multimodal visual analysis, text-based segmentation, and generative image editing. It integrates text-to-bounding-box detection and high-precision image segmentation masks to function as a text-based image segmenter and an automated visual labeling tool. The project enables text-driven image editing by identifying objects through natural language to perform inpainting and element replacement. It further extends visual analysis into three dimensions, allowing for 3D human reconstruction and the generation of 3D bounding boxes from t

Agent-S is a multimodal AI agent and LLM desktop automation framework designed to control operating systems through graphical user interface interactions. It functions as a computer use interface, utilizing vision-language grounding to translate natural language goals into precise screen coordinates and system actions. The project differentiates itself by combining structured accessibility tree inspection with vision-based element localization. It manages cross-application workflows by mapping conceptual descriptions to physical pixels and simulating low-level keyboard and mouse events to mov

Midscene is a multimodal automation framework designed to enable AI agents to perceive, navigate, and manipulate graphical user interfaces across web, mobile, and desktop environments. By leveraging vision-capable AI models, the platform interprets interface screenshots to execute tasks based on natural language instructions, removing the reliance on traditional, brittle code-based selectors. The framework distinguishes itself through its ability to decompose high-level goals into autonomous, multi-step sequences that function consistently across diverse platforms. It provides a visual ground

ml-ferret is a multimodal large language model framework and visual reasoning engine designed to reason about images and user interfaces. It functions as a UI grounding model and referring expression comprehension tool that maps natural language descriptions to precise pixel coordinates. The system focuses on high-resolution image analysis to identify and locate specific interface components. It employs multi-resolution image processing and region-aware visual encoding to preserve detail across different aspect ratios, enabling the model to analyze spatial relationships and functional layouts

Este proyecto proporciona un framework fundamental y una implementación de referencia para ejecutar modelos de lenguaje causales y razonamiento multimodal en sistemas locales. Incluye un conjunto de componentes básicos para gestionar activos de modelos, un framework de ajuste fino (fine-tuning) y las definiciones estructurales necesarias para instanciar arquitecturas basadas en transformers. El sistema se distingue por su capacidad para procesar entradas combinadas de texto e imagen a través de modelos transformer multimodales para el razonamiento visual y el análisis de documentos. También admite el despliegue de modelos cuantizados, reduciendo el uso de memoria mediante técnicas de baja precisión para permitir la inferencia en dispositivos de borde (edge devices). El proyecto cubre áreas de capacidad amplias, incluyendo el ajuste fino supervisado y la adaptación de bajo rango (LoRA) para la personalización de dominios, así como un gestor de activos integral para descargar, verificar y organizar pesos de modelos y tokenizadores. La funcionalidad adicional abarca la generación de texto multilingüe, el procesamiento de contextos largos y el grounding de lenguaje visual.

CogVLM is a multimodal large language model designed for visual reasoning and multi-turn dialogue. It functions as a visual grounding model and a quantized vision model, combining text and image processing to perform complex understanding and maintain context across visual inputs. The project includes capabilities as a GUI automation agent, allowing it to analyze application screenshots, plan operational steps, and return precise screen coordinates for interface interaction. It further supports visual grounding by generating bounding box coordinates to map text descriptions to specific spatia

DeepSeek-VL2 es un modelo de lenguaje grande multimodal y sistema de visión-lenguaje diseñado para analizar escenas visuales y generar texto descriptivo. Funciona como un modelo de respuesta a preguntas visuales y fundamentación visual (visual grounding), capaz de extraer información de documentos y localizar objetos o regiones específicas dentro de imágenes basadas en descripciones textuales. El proyecto utiliza una arquitectura de mezcla de expertos (mixture-of-experts) para procesar entradas combinadas de imagen y texto. Está optimizado para la inferencia mediante prellenado incremental, lo que reduce los requisitos de memoria de GPU en el hardware. El modelo cubre el análisis de datos multimodal y la comprensión de documentos visuales, incluyendo la interpretación de gráficos y diseños. Realiza inferencia visual y fundamentación para hacer coincidir consultas textuales con el contenido visual correspondiente.