T
TencentQQGYLab/AppAgent
AppAgent
AppAgent is an autonomous system and Android app controller that uses large language models to navigate and execute tasks within mobile applications. It functions as a mobile UI automator and element mapper, capable of performing specific application tasks by utilizing documented user interface patterns and screen navigation.
The framework differentiates itself through its ability to map application navigation and generate UI documentation via autonomous exploration or human-in-the-loop demonstrations. It employs a visual-language model to process screen screenshots and UI hierarchies to determine interaction steps.
Its broader capabilities cover mobile app automation, autonomous app exploration, and mobile UI testing. These are supported by an action-observation feedback loop and coordinate-based input simulation to translate semantic actions into device interactions.
Features
- Native Mobile App Automation - Uses large language models to execute tasks and navigate native mobile application interfaces.
- AI Agent Integrations - Integrates intelligent agents capable of perceiving mobile screens and performing multi-step actions.
- Autonomous Agent Loops - Employs autonomous planning, acting, and observing cycles to discover application navigation paths.
- Mobile App Agents - Functions as an autonomous system using large language models to navigate and execute tasks in mobile apps.
- Multimodal Large Language Models - Integrates multimodal models to process screen screenshots and UI hierarchies for determining interaction steps.
- UI Navigation Mapping - Generates documentation for user interface elements through autonomous exploration or human demonstrations.
- Android App Controllers - Provides a framework for performing application tasks on Android devices via documented UI patterns.
- Application Exploration - Maps out screens and interactive elements of mobile apps to generate navigation documentation.
- Mobile Device Automation - Identifies and interacts with mobile UI elements autonomously to map navigation and perform tasks.
- Task Execution Automations - Executes specific actions within mobile applications by using documented UI patterns to reach target screens.
- Interaction Demonstrations - Provides a mechanism to record human interactions to generate ground-truth maps for agent replication.
- Action-Observation Cycles - Implements real-time action-observation cycles to refine agent decisions based on the resulting screen state.
- Accessibility Tree Extractions - Extracts structured metadata from accessibility trees to map interactive UI elements to screen coordinates.
- Mobile Application Testing - Automates interactions with mobile application interfaces to verify the correctness of user flows.
- Coordinate-Based Interactions - Translates high-level semantic actions into precise pixel coordinates for simulating mobile device taps and swipes.
- UI Element Mapping - Generates documentation for mobile user interfaces by following human demonstrations or autonomous crawls.
- GUI Agents - Multimodal agent framework for smartphone application usage.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
