UFO

UFO² captures Windows desktop screenshots, annotates UI elements with bounding boxes and semantic labels, and executes actions (clicks, text input, keyboard commands) by mapping LLM-generated action descriptions to concrete UI coordinates. The system uses OCR and UI inspection APIs (COM-based Windows Automation Framework) to build a semantic representation of the screen state, enabling the agent to interact with any Windows application without requiring native API bindings or application-specific integrations.

UFO³ Galaxy enables a Constellation Agent to decompose high-level tasks into subtasks, distribute them across multiple registered Windows devices, and coordinate execution through an Agent Interaction Protocol (AIP). The system maintains device lifecycle state (registration, heartbeat, availability), routes tasks to appropriate devices based on capability matching, and aggregates results. Task Constellation manages task dependencies and execution order across heterogeneous devices in a network.

UFO³ provides a web-based interface for submitting automation tasks, monitoring execution progress, viewing device status, and managing device registrations. The Web UI communicates with the Galaxy orchestrator via REST APIs, displays real-time execution logs and screenshots, and allows users to pause/resume/cancel tasks. Supports role-based access control for multi-user environments.

UFO³ uses a hierarchical configuration system (YAML/JSON files) to define agent behavior, device capabilities, LLM provider settings, and knowledge base sources. Configuration files are organized by scope: agent-level (model selection, prompt templates), device-level (capabilities, resource constraints), and system-level (Galaxy settings, database connections). The system supports configuration inheritance and environment variable substitution, enabling flexible deployment across development, staging, and production environments.

UFO² includes a User Interaction Module that pauses automation and requests human input when the agent encounters ambiguous situations or needs confirmation. The module can display screenshots with annotations, ask multiple-choice questions, or request free-form text input. Responses are injected back into the agent's context, allowing it to continue with human guidance. Supports both synchronous (blocking) and asynchronous (non-blocking) interaction patterns.

UFO³ logs all execution details (actions, observations, LLM responses, tool results) to structured logs that can be analyzed for debugging and improvement. The system captures LAM (Learning from Automation Metrics) data including action success rates, LLM reasoning quality, and tool call patterns. Logs include screenshots, action traces, and full context at each step, enabling post-mortem analysis of failures. Supports log export in multiple formats (JSON, CSV) and integration with external analytics platforms.

UFO² supports both LLM-generated actions (click, type, navigate) and deterministic automation actions (MCP tool calls, COM API invocations, PowerShell scripts). The system routes actions through an Automation Framework that dispatches to appropriate executors: GUI actions go to the screenshot-annotation-action loop, while tool calls invoke registered MCP servers or COM Application Receivers. This hybrid approach allows agents to use LLM reasoning for complex UI navigation while offloading structured tasks (data extraction, API calls) to deterministic tools.

UFO² builds prompts that include desktop screenshots, extracted text (via OCR), and semantic UI annotations (element labels, bounding boxes, hierarchy). The Prompt System constructs multi-modal inputs by combining these modalities with task context and memory, then sends them to LLMs that support vision (GPT-4V, Claude 3.5). The system maintains a Prompt Component library that allows customization of how screenshots, OCR, and annotations are formatted and prioritized based on agent strategy.

UFO² implements explicit state machines for both Host Agent (window/app selection state) and App Agent (UI interaction state). Sessions represent continuous automation contexts (e.g., 'automate Excel workbook'), while Rounds represent individual LLM reasoning cycles within a session. The system tracks state transitions, maintains context across rounds, and enforces valid state progressions. Session Pool manages multiple concurrent sessions, enabling parallel automation across different applications.

UFO³ includes a RAG (Retrieval-Augmented Generation) system that allows agents to query knowledge bases (documents, FAQs, process guides) using semantic search. The system embeds documents into a vector database, retrieves relevant context based on task descriptions, and injects retrieved knowledge into prompts. Supports multiple vector database backends and allows custom knowledge creation through document ingestion pipelines.

UFO³ abstracts LLM interactions through a Service Architecture that supports OpenAI, Anthropic, Azure OpenAI, and local Ollama instances. The system handles model-specific differences (function calling schemas, vision capabilities, structured output formats) through adapter patterns. Agents can specify preferred LLM providers in configuration, and the system routes requests accordingly. Supports custom model integration through a plugin interface.

UFO³ uses structured output formats (JSON schemas, Pydantic models) to constrain LLM responses and enable reliable parsing. The system defines schemas for agent actions (click, type, navigate), task decomposition results, and tool call parameters. LLMs that support structured output (OpenAI JSON mode, Anthropic structured output) are used to generate responses matching these schemas. Responses are validated against schemas before execution, preventing malformed actions.

UFO³ integrates with MCP servers to extend agent capabilities beyond built-in actions. Agents can discover available tools from registered MCP servers, call them with structured parameters, and receive results. The system handles MCP protocol details (request/response serialization, error handling) transparently. MCP servers can be local (same machine) or remote (over HTTP/WebSocket), enabling integration with external services and tools.

UFO³ Galaxy maintains a registry of connected Windows devices with their capabilities (installed applications, available tools, resource constraints). Devices register with the Galaxy orchestrator via a registration protocol, send periodic heartbeats to signal availability, and report their capabilities. The Constellation Agent uses this capability information to route tasks to appropriate devices (e.g., 'route to device with Excel' or 'route to device with SAP access'). Device failures are detected via heartbeat timeouts, and tasks can be rerouted to healthy devices.