cua

Captures desktop screenshots and feeds them to 100+ integrated vision-language models (Claude, GPT-4V, Gemini, local models via adapters) to reason about UI state and determine appropriate next actions. Uses a unified message format (Responses API) across heterogeneous model providers, enabling the agent to understand visual context and generate structured action commands without brittle selector-based logic.

Provisions isolated execution environments across macOS (via Lume VMs), Linux (Docker), Windows (Windows Sandbox), and host OS, with unified provider abstraction. Handles VM/container lifecycle (creation, snapshot management, cleanup), resource allocation, and OS-specific action handlers (keyboard/mouse events, clipboard, file system access) through a pluggable provider architecture that abstracts platform differences.

Provides Lume provider for provisioning and managing macOS virtual machines with native support for snapshot creation, restoration, and cleanup. Handles VM lifecycle (boot, shutdown, resource allocation) with optimized startup times. Integrates with image registry for VM image management and caching. Supports both Apple Silicon and Intel Macs. Enables deterministic testing through snapshot-based environment reset between agent runs.

Provides command-line interface (CLI) for quick-start agent execution, configuration, and testing without writing code. Includes Gradio-based web UI for interactive agent control, real-time monitoring, and trajectory visualization. CLI supports task specification, model selection, environment configuration, and result export. Web UI enables non-technical users to run agents and view execution traces with HUD visualization.

Implements Docker provider for running agents in containerized Linux environments with full isolation. Handles container lifecycle (creation, cleanup), image management, and volume mounting for persistent storage. Supports custom Dockerfiles for environment customization. Provides X11/Wayland display server integration for GUI application interaction. Enables reproducible agent execution across different host systems.

Implements Windows Sandbox provider for isolated agent execution on Windows 10/11 Pro/Enterprise, and host provider for direct OS execution. Windows Sandbox provider creates ephemeral sandboxed environments with automatic cleanup. Host provider enables direct agent execution on live Windows system without isolation. Both providers support native Windows input simulation (SendInput API) and clipboard operations. Handles Windows-specific action execution (window management, registry access).

Implements comprehensive telemetry and logging infrastructure capturing agent execution metrics (latency, token usage, action success rate), errors, and performance data. Supports structured logging with contextual information (task ID, agent ID, timestamp). Integrates with external monitoring systems (e.g., Datadog, CloudWatch) for centralized observability. Provides error categorization and automatic error recovery suggestions. Enables debugging through detailed execution logs with configurable verbosity levels.

Implements the core agent loop (screenshot → LLM reasoning → action execution → repeat) via the ComputerAgent class, with pluggable callback system and custom loop support. Developers can override loop behavior at multiple extension points: custom agent loops (modify reasoning/action selection), custom tools (add domain-specific actions), and callback hooks (inject monitoring/logging). Supports both synchronous and asynchronous execution patterns.

Translates high-level action commands (click, type, scroll, key press, file operations) into OS-specific low-level operations through platform-specific handlers. Uses semantic understanding of UI coordinates and element positions to map VLM-generated actions to actual screen locations. Handles clipboard operations, file system access, and keyboard/mouse event generation with platform-specific APIs (macOS native events, Linux X11/Wayland, Windows input simulation).

Records complete agent execution traces (screenshots, actions, reasoning, timestamps) into structured trajectory files for post-execution analysis and debugging. Integrates with HUD (Heads-Up Display) system to visualize agent actions overlaid on screenshots in real-time or post-hoc. Supports trajectory export in multiple formats for benchmarking and evaluation workflows. Enables deterministic replay of agent trajectories for debugging and reproducibility testing.

Provides unified SDK interface to 100+ vision-language models across multiple providers (OpenAI, Anthropic, Google, local models via Ollama/vLLM). Supports native computer-use models (Claude with native tool use) and composed models (standard VLMs with grounding adapters that convert visual understanding to action commands). Implements provider-specific authentication, rate limiting, and error handling with fallback mechanisms. Local model adapters enable on-premise deployment without cloud API dependencies.

Tracks API token consumption and costs across VLM provider calls, with configurable budget limits and rate limiting. Implements cost estimation before execution and actual cost tracking post-execution. Supports per-agent, per-task, and global budget constraints with automatic throttling or termination when limits are exceeded. Integrates with provider-specific pricing models (OpenAI, Anthropic, Google) for accurate cost calculation.

Provides evaluation infrastructure for agent performance assessment using standardized benchmarks (OSWorld, etc.). Implements evaluation workflows that execute agents on benchmark tasks, collect trajectories, and compute metrics (success rate, cost per task, steps to completion). Integrates with OSWorld benchmark suite for comparative evaluation. Supports custom evaluation metrics and task definitions. Generates evaluation reports with detailed performance breakdowns.

Provides parallel SDKs in Python (cua-agent, cua-computer) and TypeScript (cua-agent, cua-computer) with unified API design enabling developers to write agent code in their preferred language. Both SDKs expose ComputerAgent, Computer, and execution environment classes with identical method signatures and behavior. Supports both synchronous and asynchronous execution patterns. Includes CLI tools for quick-start and testing.

Implements MCP server support enabling agents to call external tools and services through standardized MCP protocol. Allows developers to expose custom tools, APIs, and services as MCP resources that agents can discover and invoke. Supports both built-in tools (file operations, web search) and custom tools via MCP server registration. Handles tool discovery, invocation, and result integration into agent reasoning loop.