mobile-next/mobile-mcp
Mobile Mcp
This project is a Model Context Protocol server and automation framework designed to control and automate iOS and Android devices. It provides a unified API that abstracts interactions between physical hardware and simulators across different mobile operating systems, functioning as a cross-platform device bridge.
The system is distinguished by a visual UI automation toolkit that uses screenshots and coordinate-based gestures—such as tapping, swiping, and long-pressing—rather than relying on element selectors. It supports remote connectivity via an HTTP server using Server-Sent Events, which can be secured with bearer token authorization.
The framework covers a broad range of capabilities, including application lifecycle management, device discovery, and hardware button simulation. It also includes tools for screen state recording, UI accessibility inspection, and the resolution of CAPTCHA challenges through visual analysis.
Features
- Mobile Device Automation - Provides a unified framework for automating native applications across both iOS and Android platforms.
- Cross-Platform Device Managers - Provides a unified API that abstracts interactions across iOS and Android devices, functioning as a cross-platform device bridge.
- Model Context Protocol Servers - Exposes mobile device control tools as a standardized Model Context Protocol server for AI agent integration.
- Screen Capture Tools - Takes screenshots of the current display for visual content analysis and UI state determination.
- Device Connection Management - Establishes and manages connections to physical mobile devices, simulators, and emulators through platform SDKs.
- Device Enumeration - Provides the ability to discover and list all connected iOS and Android devices and emulators.
- Application Lifecycle Management - Provides comprehensive control over the mobile application lifecycle, including installation, launching, and termination.
- Visual State Analysis - Captures screen content for LLM-driven interpretation of UI states and CAPTCHA challenges.
- Cross-Platform Driver Bridges - Implements a unified API that translates high-level automation commands into platform-specific actions for iOS and Android.
- Coordinate-Based Interactions - Implements UI automation by targeting specific pixel coordinates for taps and swipes without needing element selectors.
- Element Enumerations - Lists all visible user interface elements along with their coordinates and accessibility properties for automation mapping.
- Visual UI Automation Toolkits - Provides a toolkit for capturing screenshots and analyzing coordinates to drive LLM-based mobile interactions.
- SSE Transports - Exposes MCP tools over HTTP using Server-Sent Events for bidirectional streaming connectivity.
- Structured Data Extraction - Extracts structured information from device screen content, typically for LLM-driven interpretation of the UI state.
- Mobile Application Deployments - Provides the ability to install application package files from a local path onto connected mobile devices.
- UI Element State Recording - Records the visual state and lists UI elements with their associated coordinates and properties.
- Installed Package Enumeration - Retrieves a complete list of all applications currently installed on a device by their package names.
- Workflow Automations - Executes complex, multi-step sequences of interactions to automate tasks like account registration.
- Headless Emulator Control - Controls mobile simulators or emulators that operate without a physical display attached.
- UI State Inspection - Analyzes accessibility trees and screenshots to map the interface of mobile applications.
- Screen Metadata Retrieval - Returns the current pixel size and orientation of the connected mobile device screen.
- SSE Remote Access - Ships an HTTP server using Server-Sent Events to enable secure remote access to device automation tools.
- Hardware Button Simulations - Simulates physical presses of device hardware buttons such as Home, Back, and Volume.
- Automated Captcha Solvers - Analyzes screenshots of captcha windows to programmatically solve challenges during automation.
- Process Termination Utilities - Implements functionality to stop and close specific active applications on a mobile device.
- Application Uninstallers - Provides utilities to remove applications from a device using unique bundle IDs or package names.
- Mobile Device Discovery - Retrieves a list of all connected simulators and physical devices for target selection.
- App Process Starters - Allows starting specific applications on a device using bundle or package identifiers.
- Device Orientation Controllers - Programmatically rotates the device screen between portrait and landscape modes to verify layouts.
- Input Simulations - Mimics human interaction by programmatically simulating text typing and hardware button presses.
- Screen Swipe Simulations - Executes automated swipe gestures across the screen in specified directions.
- Text Input Simulations - Automates text entry into the currently focused input element on the mobile device.
Star history
How this analysis was created: This summary and feature list are AI-generated from collected project material and can contain mistakes. Stars, license and language are imported from GitHub. Inclusion does not mean that we have tested or audited this project. Check the source documentation for any feature you depend on. Learn more on our About page.
