bytedance/UI-TARS
UI TARS
UI-TARS is an LLM GUI automation framework and multimodal action grounding system. It functions as a GUI agent orchestrator and cross-platform device controller that uses large language models to interpret graphical interfaces and execute actions across desktop and mobile operating systems.
The system translates model-generated coordinates into precise screen positions to interact with visual user interface elements. It employs a multimodal approach to interpret screen layouts and decomposes complex goals into multi-step trajectories through reasoning and error correction.
The project provides capabilities for cross-platform interface control, including clicking, typing, and scrolling across web, mobile, and desktop environments. It includes tools for desktop and mobile GUI interaction, automation script generation, and visual grounding evaluation to measure coordinate precision.
The framework supports hosting models on cloud platforms to provide scalable inference endpoints.
Features
- Autonomous Agent Orchestrators - Decomposes complex goals into multi-step trajectories and manages contextual memory for autonomous interface navigation.
- Multimodal Vision Interfaces - Integrates vision-capable models to process screenshots and text prompts for interpreting the current state of graphical user interfaces.
- Automated Desktop Interaction Systems - Performs mouse clicks and keyboard shortcuts to navigate desktop browsers and files using visual analysis.
- Coordinate Normalization Utilities - Translates relative model output coordinates into absolute pixel positions based on target screen resolution for visual interaction.
- Cross-Platform GUI Controllers - Translates structured instructions into native clicks, typing, and scrolling events across web, mobile, and desktop environments.
- Action Data Normalization - Converts generated action strings into organized data sets by scaling coordinates and normalizing formats for various model types.
- LLM GUI Automation Frameworks - Uses large language models to interpret graphical interfaces and execute actions across desktop and mobile operating systems.
- Multi-step Goal Execution - Decomposes complex goals into sequential trajectories using high-level reasoning and error correction.
- Multimodal Interaction Engines - Bridges visual perception with language models to map generated coordinates to precise screen positions.
- Text-to-Coordinate Mapping - Maps extracted screen text and visual elements to precise pixel coordinates to enable targeted automated interactions.
- Real-Time GUI Interpretation - Processes multimodal inputs including text and images to understand and monitor dynamic user interfaces in real-time.
- Task Planning Systems - Decomposes complex user goals into a sequence of discrete actions using a loop of reasoning and error correction.
- GUI Agents - Operates as a multimodal agent capable of interpreting screen layouts and performing multi-step tasks across desktop and mobile interfaces.
- GUI Action Parsing - Converts natural language model responses into machine-readable data sets for the execution of GUI interactions.
- Cross-Platform Desktop Automation Libraries - Executes standardized interaction primitives like clicking, typing, and scrolling across desktop, mobile, and web environments.
- Screen Space Coordinate Mappings - Converts normalized 2D model coordinates into absolute pixel positions based on current target screen dimensions.
- Native Mobile Automation - Controls native mobile applications and emulators through automated gestures and navigation to simulate user behavior.
- Desktop Automation - Automates workflows within a desktop environment by performing clicks, drags, and keyboard shortcuts in office software.
- Cross-Platform Abstraction Layers - Provides a unified interface to execute standardized interaction commands across web, mobile, and desktop environments.
- Hybrid Short-and-Long Term Memory - Maintains a hybrid system of short-term task context and long-term interaction history to improve agent decision-making.
- Visual Grounding Execution - Maps specific coordinates to visual elements by outputting direct actions without reasoning steps to evaluate precision.
- Visual Grounding - Measures model precision by mapping coordinate outputs to specific visual elements on a screen for grounding evaluation.
- Model Action Runtimes - Translates language model text responses into structured data for execution within an automation runtime.
- GUI - Converts structured action data into executable code for automating GUI tasks like clicking and dragging.
- GUI Automation Libraries - Simulates user interactions like clicking and typing across interfaces to verify software behavior and performance.
- GUI and Computer Agents - Automated GUI interaction using native agents.
- Specialized Multimodal Agents - Native agent model for automated GUI interaction via screenshot perception.
