microsoft/UFO
UFO
UFO is a multi-device task orchestrator and LLM agent orchestration framework designed to decompose natural language requests into executable task graphs. It functions as a cross-platform UI automation tool capable of performing interactions on Windows and mobile devices while routing tasks to distributed agents based on their hardware and software capabilities.
The system is distinguished by its RAG-enhanced agent architecture, which integrates external documentation and previous execution traces to improve decision-making. It employs a hybrid UI detection approach that combines computer vision with accessibility frameworks to identify on-screen elements more reliably than standard automation tools.
The framework covers a broad range of capabilities including distributed task scheduling, remote device management, and dynamic workflow orchestration. It manages the entire lifecycle of a request, from translating high-level goals into dependent subtasks to monitoring connection health via heartbeats and visualizing execution flows in real-time.
The project is implemented in Python and includes a toolkit for creating custom device agents and defining device registries via configuration files.
Features
- Multi-Agent Orchestration - Orchestrates the decomposition and delegation of complex natural language requests across multiple distributed agents.
- Multi-Agent Orchestrators - Coordinates specialized AI agents across diverse operating systems to solve complex user requests.
- Multi-Agent Task Orchestrators - Provides a coordination layer for managing complex, multi-step workflows across diverse AI agents on multiple devices.
- Agent Definitions - Defines agent identities, endpoints, and operating systems via configuration files to establish a device registry.
- Agent Orchestration Frameworks - Decomposes natural language requests into executable task graphs for distributed agent execution.
- Agent Registrations - Onboards devices by defining identities, capabilities, and endpoints through a multi-phase registration process.
- Custom Agent Builders - Provides a template-driven toolkit for creating custom device agents with support for tool augmentation.
- Capability-Based Routing - Routes tasks to devices based on hardware metadata and functional tags to ensure necessary tools are available.
- Cross-Platform Visual Automation Tools - Performs touch events and GUI interactions on mobile and Windows devices using visual recognition.
- Hybrid UI Detection Systems - Combines computer vision and accessibility framework data to reliably identify on-screen elements for automation.
- Retrieval-Augmented Agents - Integrates external documentation and historical execution traces into the agent decision process using RAG.
- Specialist Task Routing - Assigns jobs to specific agents by matching required functional capabilities and hardware metadata.
- Task Planning Systems - Translates high-level goals into a structured graph of tasks and dependencies mapped to specific devices.
- Natural Language Automation - Translates high-level natural language prompts into structured task graphs and device assignments.
- Capability-Based Assignments - Matches tasks to the most suitable device based on platform, resource monitoring, and performance history.
- Device-Capability Matching - Assigns tasks to the most suitable device based on platform capabilities, resource monitoring, and performance history.
- Natural Language Automation - Translates high-level natural language requests into executable automation workflows for specific target devices.
- Device Capability Registrations - Assigns functional tags to devices to enable intelligent task routing based on available tools and features.
- Hybrid Layer Execution - Combines graphical user interface interaction with API calls to execute complex workflows across different software layers.
- Distributed Task Schedulers - Routes asynchronous jobs to remote devices based on hardware capabilities and current status.
- Task Distribution Strategies - Distributes tasks using round-robin or load-balancing strategies while tracking device status and validation.
- Native Mobile Automation - Performs native touch events, typing, and application launches on mobile devices to automate user interface interactions.
- Distributed Device Orchestration - Orchestrates automated workflows across different operating systems based on device hardware capabilities.
- Dynamic Task Graphs - Decomposes natural language goals into a dynamic graph of interdependent subtasks that can be modified during runtime.
- Complex Workflow Coordination - Coordinates dependent execution chains and groups of subtasks assigned to specific devices for complex goal achievement.
- Multi-Device Management - Unifies connection lifecycles and task routing across a distributed network using specialized orchestrator and executor endpoints.
- Cross-Platform Workflow Coordination - Decomposes complex requests into dynamic graphs to coordinate task execution across various operating systems and web platforms.
- Hybrid Element Detection - Identifies on-screen controls by combining visual detection with automation framework data for increased robustness.
- Cross-Platform UI Automations - Provides automation capabilities for software interfaces on Windows and mobile devices using computer vision.
- UI Element Selectors - Locates and queries UI elements using accessibility frameworks and computer vision for navigational purposes.
- Agent Communication Interfaces - Uses a unified WebSocket-based interaction protocol for secure and fault-tolerant coordination between agents and the orchestrator.
- Agent Communication Protocols - Implements a standardized communication layer for agent registration, heartbeats, and task dispatching.
- Agent Knowledge Bases - Augments agent decision making by integrating external documents and historical execution trajectories.
- Execution Flow Visualization - Renders task graphs in multiple modes to track real-time progress and debug distributed operations.
- Action Sequence Prediction - Combines multiple action predictions into a single sequence to minimize external model calls and increase execution efficiency.
- Dynamic Workflow Modification - Enables the dynamic rewiring of task graphs and dependency edges during execution to adapt to evolving requirements.
- Checkpoint-Based State Restoration - Resumes agent state and task continuity after network disconnections to prevent cascade failures.
- Server Capability Discovery - Uses a standardized protocol to discover and route commands to specialized servers for task manipulation.
- Task Execution Sequencing - Coordinates tasks into sequential or parallel execution paths while respecting complex dependency constraints across devices.
- Task Dependency Management - Defines execution order and prerequisites for tasks using conditional logic to control workflow flow.
- Finite State Tracking - Tracks the progress of a task through a finite state machine to determine if execution should continue or fail.
- API-Driven Task Triggers - Initiates remote execution and monitoring of tasks on client devices via a web API.
- Heartbeat Monitors - Monitors remote agent health using periodic heartbeat signals to trigger automatic reconnection and state restoration.
- Agent Communication Protocols - Implements a persistent connection protocol to ensure secure, fault-tolerant communication between the orchestrator and distributed agents.
- Device Communication Protocols - Provides a consistent protocol for device registration, system telemetry, and real-time task progress reporting.
- Websocket Connection Managers - Establishes persistent WebSocket connections using unique IDs and monitors health status via periodic heartbeats.
- Connection Management - Monitors session stability using heartbeats and reconnection strategies to ensure operational continuity.
- Windows Desktop Automation - Automates the Windows user interface, applications, and system functions using native UI automation and system calls.
- Device and Environment Metadata - Combines static configurations with service manifests and real-time telemetry to create comprehensive device profiles.
- Asynchronous Task Execution - Executes long-running operations across different devices via durable sessions with real-time log streaming.
- Finite State Machine Engines - Tracks the lifecycle and progress of remote tasks using a finite state machine to manage failures and state.
- Graph Validation - Checks task graphs for structural errors and circular dependencies to ensure a valid execution order.
- Graph Consistency Enforcement - Enforces structural invariants within the task graph to prevent race conditions during active execution.
- Session Coordination - Manages the lifecycle of requests by initializing sessions, triggering task decomposition, and aggregating results.
- Task Execution Monitoring - Tracks task progress and execution graphs via a real-time web interface and chat.
- Device State Collection - Captures screenshots, UI trees, and application lists from devices to provide necessary visual and structural context.
- Execution Logging and Diagnostics - Records task decomposition and captures screenshots to analyze routing decisions and results.
- Remote Device Management - Provides centralized onboarding and monitoring for a fleet of remote agents via secure connections.
- Status Monitors - Monitors agent operational state including connection health and heartbeat timestamps in real-time.
- AI Agents - Windows desktop automation agent for multi-application tasks.
- GUI Agents - UI-focused agent for Windows operating system interaction.
- Task Automation Agents - UI-focused agent framework for navigating and operating Windows applications.
