UI-TARS-desktop

Orchestrates multimodal AI agents through a ComposableAgent plugin architecture that dynamically chains GUI, code, MCP, and browser automation tools. Implements a T5 format streaming parser for structured LLM output and a Tarko framework execution loop that manages agent state, tool invocation, and event streaming. Agents receive vision-language model outputs (screenshots, structured data) and route them through specialized plugin handlers that execute actions and feed results back into the reasoning loop.

Automates desktop and web UI interactions by capturing screenshots, sending them to a vision-language model (VLM), parsing structured action commands (click, type, scroll), and executing them via the GUIAgent SDK. The SDK provides operator implementations for local (Electron-based) and remote (VNC/RDP) desktop control, with coordinate-based action execution and screen state feedback loops. Supports both UI-TARS proprietary models (Doubao-1.5-UI-TARS) and generic vision LLMs through a configurable VLM provider interface.

Implements a hooks and lifecycle event system that allows custom code to execute at specific points in the agent execution loop (before/after tool call, on error, on completion). Hooks are registered at agent initialization and invoked by the Tarko framework during execution, enabling extensibility without modifying core agent code. Events include reasoning, tool_call, result, error, and completion, with detailed context passed to hook handlers.

Provides runtime settings management that allows agents to be reconfigured without restart, including tool registration, model parameters, execution timeouts, and resource limits. Settings are stored in a configuration object that can be updated via REST API or programmatically, with changes taking effect immediately for new tool invocations. Supports per-session and global settings with hierarchical override (session > global).

Implements the core agent execution loop (Agent Runner) that orchestrates reasoning, tool invocation, and result feedback in an iterative cycle. The loop executor manages execution state, handles streaming output from the LLM, invokes tools via the tool call engine, and feeds results back into the next reasoning step. Supports configurable loop termination conditions (max iterations, tool completion, explicit stop) and provides detailed execution traces for debugging.

Implements a tool call engine that validates tool invocations against registered tool schemas, handles tool execution via multiple strategies (direct function call, MCP server, subprocess), and manages tool result formatting. The engine supports tool retries on failure, timeout handling, and error recovery. Tool execution strategies are pluggable, allowing custom implementations for specific tool types (e.g., subprocess for shell commands, MCP for remote tools).

Provides a content rendering system that formats agent outputs (text, code, images, structured data) for display in the web UI or other frontends. Supports rendering of code blocks with syntax highlighting, images with metadata, structured data as tables or JSON, and markdown-formatted text. The rendering system is extensible, allowing custom renderers for specific content types.

Integrates Model Context Protocol (MCP) servers as dynamically registered tools within the agent framework, using an MCP client architecture that handles transport (stdio, SSE, WebSocket), schema discovery, and tool invocation. The MCP Agent Plugin wraps MCP server capabilities into the ComposableAgent plugin interface, automatically discovering tool schemas and mapping them to the T5 format for LLM tool calling. Supports multiple concurrent MCP server connections with isolated resource management and error handling per server.

Provides browser automation infrastructure for agents to control headless browsers (Chromium via Puppeteer/Playwright), capture DOM state, execute JavaScript, and interact with web pages. Integrates a search system layer that enables agents to perform web searches (via configurable search providers) and navigate results. The browser control layer abstracts page navigation, element interaction, and screenshot capture, feeding visual and DOM state back into the agent reasoning loop for next-step decisions.

Provides a Code Agent plugin that executes arbitrary code (Python, JavaScript, shell) in isolated sandbox environments, capturing stdout/stderr and execution results. Integrates with the Tarko framework to manage sandbox lifecycle, handle timeouts, and return execution results to the agent reasoning loop. Supports both local execution (for development) and remote sandbox services (for production isolation), with configurable resource limits and execution timeouts.

Implements a T5 format streaming parser that converts LLM output (from vision-language models) into structured tool calls and reasoning traces. The parser handles partial/incomplete streaming responses, validates tool schemas against registered tools, and emits parsing events (tool_call, reasoning, error) that feed into the agent execution loop. Supports recovery from malformed output and provides detailed error messages for debugging LLM output issues.

Manages agent session lifecycle (creation, execution, termination) through the Tarko Agent Server framework, which provides REST endpoints for session creation, query submission, and event streaming. Sessions maintain state (agent configuration, tool registry, execution history) and emit events (tool_call, reasoning, result, error) that are streamed to clients via Server-Sent Events (SSE) or WebSocket. Event storage persists execution history for audit, debugging, and session resumption.

Provides a web-based UI (Tarko Agent Web UI) for configuring and composing agents without code, allowing users to select agent type (OmniTARS, GUI Agent, Code Agent), choose LLM provider and model, register tools (MCP servers, browser, code sandbox), and set runtime parameters. Configuration is serialized as JSON and passed to the agent server, enabling dynamic agent composition at runtime. The UI includes workspace navigation, session history, and content rendering for agent outputs.

Packages UI-TARS as a native Electron desktop application that provides local GUI automation (via GUIAgent SDK) and remote desktop control (via VNC/RDP). The Electron main process handles system permissions (screenshot, input simulation), manages local browser/sandbox processes, and communicates with remote desktop servers. The renderer process provides a React-based UI for configuration, session management, and real-time visualization of agent actions on the desktop.

Abstracts vision-language model (VLM) providers through a configurable interface that supports OpenAI-compatible APIs, Anthropic Claude, and proprietary UI-TARS models (Doubao-1.5-UI-TARS). The VLM provider layer handles API authentication, request formatting, streaming response parsing, and error handling. Agents can switch between VLM providers at runtime by changing configuration, enabling model comparison and fallback strategies.