droidrun/droidrun
Droidrun
Droidrun is a mobile device automation framework that uses large language models to translate natural language commands into executable actions on mobile operating systems. It functions as an agent orchestrator and UI automation engine, providing a reasoning engine that decomposes complex mobile tasks into smaller, manageable steps.
The system distinguishes itself through a hierarchical action translation process and the ability to analyze accessibility trees and screenshots to determine the visual layout and current status of mobile applications. It supports execution across both physical hardware and managed remote cloud environments, removing the requirement for local hardware setup.
The framework covers several capability areas, including structured data extraction from native mobile applications using natural language queries and telemetry-driven execution tracing to visualize agent decision paths. It also provides a system for extending agent functionality through custom tools and specialized guidance.
Features
- Natural Language Command Translation - Translates natural language commands into executable device-level actions on mobile operating systems using LLMs.
- Natural Language Device Control - Uses large language models to translate natural language commands into direct mobile device actions.
- AI Agent Orchestration - Coordinates specialized agents combining custom instructions and tools for complex mobile workflows.
- AI Agent Orchestrators - Provides a reasoning engine that decomposes complex mobile tasks into traceable and debuggable steps.
- Multi-step Goal Execution - Plans and executes sequences of actions by breaking large tasks into smaller, manageable steps.
- Agentic Goal Decomposition - Uses a planning loop to break complex user goals into smaller executable steps via LLMs.
- UI State Analysis - Analyzes accessibility trees and screenshots to determine the current state and visual layout of a mobile app.
- Android Device Controllers - Provides a controller that executes automation scripts across physical Android hardware and cloud environments.
- UI State Inspection - Analyzes accessibility trees and screenshots to determine the visual layout and current status of a mobile application.
- Native Mobile Automation - Inspects accessibility trees and screenshots to retrieve structured data and control native mobile applications.
- Accessibility-Tree Addressing - Maps application UI hierarchies and screenshots into structured data for agent reasoning and interaction.
- Native Mobile App Automation - Executes sequences of actions on mobile devices by translating natural language instructions into device controls.
- Agent Execution Tracing - Logs and visualizes the decision-making path of an agent to debug its operational logic.
- Agent Capability Extensions - Integrates custom external tools and specialized guidance to expand the functional range of the agent.
- Mobile UI Data Extraction - Retrieves structured information from native mobile applications using natural language queries.
- Hardware-Targeted Remote Execution - Runs automation scripts in managed cloud environments via a remote connection to mobile hardware.
- Script Execution Environments - Provides a managed remote environment for executing mobile app scripts without local hardware.
- UI Content Extraction - Retrieves structured information from native mobile application user interface elements.
- Mobile Testing Platforms - Provides virtualized mobile devices and cloud environments for running automation scripts.
