mobile-mcp

Provides a single Robot interface abstraction layer that normalizes interactions across Android (physical devices and AVD emulators), iOS (physical devices via USB), and iOS Simulators (via xcrun simctl). The architecture uses platform-specific manager implementations (AndroidRobot, IosRobot, SimctlManager) that all conform to a common Device API contract, eliminating the need for agents to understand platform-specific tool invocation patterns. Device resolution is request-scoped and stateless, with each tool call resolving the target device parameter through getRobotFromDevice() to the appropriate platform manager.

Extracts and parses native accessibility trees from both Android (via ADB accessibility service) and iOS (via WebDriverAgent accessibility API) to enable deterministic, coordinate-free UI interaction. The system builds a hierarchical representation of UI elements with semantic labels, roles, and bounds, allowing agents to locate and interact with elements by accessibility properties rather than fragile pixel coordinates. Falls back to screenshot-based coordinate tapping only when accessibility data is unavailable, providing a two-tier interaction strategy that prioritizes semantic stability.

Manages WebDriverAgent session lifecycle for iOS devices (both physical and simulators) including session creation, teardown, and error recovery. The WebDriverAgent client (src/webdriveragent.ts) handles HTTP communication with WebDriverAgent endpoints, session initialization with app bundle IDs, and timeout management. The system maintains session state per device and automatically re-establishes sessions on failure. Session management is abstracted from agents — they invoke Robot interface methods without understanding WebDriverAgent protocol details. The implementation handles both localhost communication (simulators) and USB tunnel communication (physical devices) transparently.

Provides image processing utilities for screenshot analysis, including screenshot capture, image format conversion, and visual element detection support. The system captures screenshots from devices through platform-specific mechanisms (ADB screencap for Android, WebDriverAgent screenshot API for iOS) and processes them through image utilities for format conversion and metadata extraction. The implementation supports PNG and JPEG formats and provides hooks for visual element detection (though advanced CV/ML-based detection is not built-in). Screenshots are used as fallback when accessibility tree data is unavailable and for visual validation workflows.

Provides app installation, launch, termination, and state management capabilities across Android and iOS platforms. On Android, app lifecycle is managed through ADB commands (adb install, adb shell am start, adb shell am force-stop). On iOS, app lifecycle is managed through go-ios (for physical devices) and simctl (for simulators). The system supports app installation from APK/IPA files, launching apps with intent/URL parameters, and force-stopping/terminating apps. App state is managed per device, allowing agents to control app lifecycle as part of automation workflows.

Captures full-screen screenshots from the device and enables coordinate-based interaction (tap, swipe, drag) when accessibility tree data is unavailable or insufficient. The system processes screenshots through image processing utilities to extract visual information, then maps agent-specified coordinates or visual regions to device touch events. This provides a fallback mechanism for apps with poor accessibility implementation or for visual-based automation scenarios where semantic interaction is not viable.

Implements AndroidRobot class that wraps Android Debug Bridge (ADB) for controlling physical Android devices and AVD emulators. The implementation handles ADB command execution, device state management, accessibility service integration for UI tree extraction, and gesture simulation (tap, swipe, long-press) through ADB input events. Device discovery and management is handled by AndroidDeviceManager, which enumerates connected devices via 'adb devices' and maintains device-specific state. The architecture abstracts ADB complexity behind the common Robot interface, allowing agents to control Android devices without direct ADB knowledge.

Implements IosRobot class that controls iOS physical devices (iPhone, iPad) connected via USB using the go-ios tool for device communication and WebDriverAgent for UI automation. The architecture uses go-ios for low-level device operations (device discovery, app installation, log streaming) and WebDriverAgent (a native iOS testing framework) for UI interaction and accessibility tree extraction. Device management is handled by IosManager, which discovers connected iOS devices via go-ios and maintains WebDriverAgent session state. The implementation abstracts the complexity of USB tunneling, WebDriverAgent session management, and iOS-specific constraints behind the common Robot interface.

Implements SimctlManager and simulator-specific Robot implementation that controls iOS Simulators using xcrun simctl (Xcode's simulator control tool) for device management and WebDriverAgent for UI automation. The architecture uses simctl to enumerate running simulators, launch/terminate simulator instances, and manage simulator state, while WebDriverAgent provides UI interaction and accessibility tree extraction. Unlike physical iOS devices, simulators run locally and communicate with WebDriverAgent over localhost, eliminating USB tunnel complexity. The implementation abstracts simctl command execution and WebDriverAgent session management behind the common Robot interface.

Provides device discovery and management across all supported platforms through platform-specific managers (AndroidDeviceManager, IosManager, SimctlManager) that enumerate connected/running devices and resolve device identifiers to Robot instances. The system maintains a registry of available devices and their capabilities, enabling agents to query device lists, filter by platform/type, and dynamically resolve device parameters at invocation time. Device resolution is request-scoped and stateless, allowing horizontal scaling and multi-device orchestration without persistent device state management. The architecture supports mixed-platform automation (Android + iOS in same workflow) through unified device resolution.

Provides cross-platform gesture simulation (tap, swipe, long-press, drag, multi-touch) by translating high-level gesture specifications into platform-specific input events. On Android, gestures are implemented via ADB input event commands (sendevent for low-level events or input tap/swipe for high-level commands). On iOS, gestures are implemented through WebDriverAgent's gesture API. The system supports both coordinate-based gestures (when accessibility data is unavailable) and element-based gestures (when accessibility tree provides element bounds). Gesture parameters (duration, velocity, pressure) are normalized across platforms to provide consistent behavior.

Implements a Model Context Protocol (MCP) server that exposes mobile automation capabilities as MCP tools, enabling LLM clients and AI agents to invoke mobile automation through the standardized MCP protocol. The server (src/server.ts) registers tools for device discovery, UI interaction, screenshot capture, and gesture simulation, mapping MCP tool schemas to Robot interface methods. The implementation supports multiple MCP transport modes (stdio, SSE) and handles tool invocation, parameter validation, and error reporting through the MCP protocol. The server is stateless and request-scoped, allowing multiple concurrent clients to orchestrate different devices without session conflicts.

Implements a cross-platform error handling strategy that catches platform-specific errors (ADB connection failures, WebDriverAgent session timeouts, simctl command failures) and translates them into standardized error responses through the MCP protocol. The system includes device state recovery mechanisms such as automatic WebDriverAgent session re-establishment on iOS, ADB reconnection on Android, and simulator state validation. Error handling is integrated into each platform manager (AndroidRobot, IosRobot, SimctlManager) and propagated through the Robot interface to the MCP server layer, providing consistent error reporting to agents.