MobileAgent

Uses GUI-Owl vision-language models (1.5, 7B, 32B variants) built on Qwen3-VL to perform native visual understanding of mobile/desktop UI elements and generate precise bounding box coordinates for detected components. The model unifies perception, grounding, and reasoning in a single forward pass, enabling pixel-accurate element localization without separate object detection pipelines or post-processing heuristics.

Implements hierarchical task planning using GUI-Owl reasoning capabilities to decompose high-level user intents into sequences of atomic GUI actions (tap, swipe, type, scroll). The framework uses explicit thinking chains (Thinking variants of GUI-Owl) to generate step-by-step action plans with intermediate state validation, enabling recovery from partial failures and dynamic replanning when UI state diverges from expectations.

Provides comprehensive evaluation framework with standardized benchmarks (GroundingBench, GUIKnowledgeBench) to measure agent performance on mobile automation tasks. Metrics include action success rate, task completion rate, action efficiency (steps to completion), and grounding accuracy. Enables reproducible comparison across agent versions and model variants.

Accepts high-level natural language task descriptions (e.g., 'send a message to John saying hello') and uses GUI-Owl reasoning to understand user intent, extract key entities and constraints, and map them to concrete automation objectives. Handles ambiguous or incomplete specifications by asking clarifying questions or making reasonable assumptions based on app context.

Maintains a rolling history of executed actions, screenshots, and outcomes to provide context for planning and reflection. Uses this history to detect patterns (repeated failures, circular action sequences), identify state divergence from expected trajectory, and inform replanning decisions. Implements efficient history compression to manage memory usage in long-running automations.

Provides a unified action execution layer that translates high-level GUI actions (tap, swipe, type, scroll) into platform-specific commands via pluggable controllers: AndroidController (ADB), HarmonyOSController (HarmonyOS APIs), PyAutoGUI (desktop), and Playwright (browser). Each controller implements a common interface, enabling the same action plan to execute across mobile and desktop without modification.

Captures post-action screenshots and uses GUI-Owl perception to validate whether the executed action achieved its intended effect (e.g., confirming a button press changed the UI state). Implements a feedback loop that detects action failures (element not clickable, network timeout) and triggers replanning or retry logic, enabling self-correcting automation without explicit error handling code.

Implements UI-S1 training pipeline using VERL framework to fine-tune GUI-Owl models on real mobile app interactions through semi-online RL. The system collects trajectories from live app executions, generates synthetic rewards based on task completion and action efficiency, and updates the model to improve action selection without requiring manual annotation. Enables continuous improvement of automation policies as new app versions and UI patterns are encountered.

Implements GUI-Critic-R1 module that analyzes planned action sequences before execution to predict and diagnose potential failures (unreachable elements, invalid state transitions, missing prerequisites). Uses extended reasoning to evaluate action feasibility against current UI state and generates diagnostic reports with suggested corrections, reducing failed executions and improving overall automation reliability.

Extends Mobile-Agent framework to desktop (Windows/macOS/Linux) and web browsers through PC-Agent and Playwright-based controllers. Implements platform-specific element detection (Windows UI Automation, macOS Accessibility APIs, DOM parsing for web) and action execution (pywinauto, macOS native APIs, Playwright commands), enabling unified automation across mobile, desktop, and web with minimal code changes.

Mobile-Agent-E implements self-evolution mechanism where the agent learns new capabilities and refines existing ones through interaction with diverse apps and user feedback. The system maintains a capability registry, collects execution traces, and uses reinforcement learning to expand the action vocabulary and improve decision-making for novel UI patterns not seen during initial training.

Mobile-Agent-v2 implements multi-agent system where specialized agents handle different aspects of automation: planning agent decomposes tasks, execution agent performs actions, reflection agent validates outcomes and triggers replanning. Agents communicate through shared state (screenshots, action history) and coordinate via a central orchestrator that manages task flow and error recovery.

Maintains a knowledge base of common UI patterns, element types, and interaction semantics across diverse apps. GUI-Owl models leverage this knowledge during perception and planning to understand element purpose (button, input field, navigation) and predict likely interactions, improving grounding accuracy and action selection without requiring app-specific training.