@github/computer-use-mcp

Captures full-screen or region-specific screenshots from the host desktop and returns pixel-perfect image data in base64 format, enabling AI agents to visually perceive and analyze the current UI state. Integrates with native OS screenshot APIs (macOS/Linux/Windows) through Node.js bindings, providing sub-100ms capture latency for real-time visual feedback loops in agent decision-making.

Enables precise mouse cursor positioning and click operations (single-click, double-click, right-click) at specified screen coordinates, translating high-level agent intents into low-level input events. Uses native OS input APIs (Xdotool on Linux, CGEvent on macOS, SendInput on Windows) to simulate human-like mouse interactions with configurable timing and movement curves to avoid detection as automated input.

Maintains a detailed audit trail of all operations performed by agents, including operation type, parameters, timestamp, and result. Logs are stored locally and can be retrieved through MCP interface for debugging, compliance, or workflow analysis. Implements structured logging with configurable verbosity levels and optional sensitive data redaction for security-sensitive operations.

Simulates keyboard input including text typing, individual key presses, and multi-key hotkey combinations (Ctrl+C, Cmd+Z, etc.) at the OS level. Implements key event queuing with configurable inter-key delays to simulate human typing speed, and supports modifier key combinations for application shortcuts. Routes through native OS keyboard APIs to ensure compatibility with applications that validate input source.

Implements a complete MCP (Model Context Protocol) server that exposes computer-use capabilities as standardized MCP resources and tools, enabling any MCP-compatible client (Claude, custom agents, etc.) to discover and invoke desktop automation functions. Uses JSON-RPC 2.0 transport over stdio or network sockets, with automatic capability advertisement through MCP's resource and tool schemas.

Detects and handles multiple physical monitors and virtual display configurations, allowing agents to capture screenshots and perform interactions across the entire display landscape. Maintains a coordinate system that maps logical screen positions to physical monitor positions, enabling agents to work with multi-monitor setups without explicit monitor selection. Automatically detects display topology changes and updates coordinate mappings.

Enumerates open application windows on the desktop and provides window focus control, allowing agents to switch between applications and ensure keyboard/mouse input targets the correct window. Returns window metadata including title, process ID, window bounds, and focus state. Implements platform-specific window management (wmctrl on Linux, NSWindow API on macOS, Windows API on Windows) with a unified interface.

Provides read and write access to the system clipboard, enabling agents to exchange text data with applications through copy/paste operations. Implements platform-specific clipboard APIs (xclip on Linux, NSPasteboard on macOS, Windows Clipboard API) with support for both text and rich text formats. Allows agents to retrieve clipboard contents for verification or use clipboard as a data exchange mechanism.

Detects and reports system information including OS type/version, available displays, installed applications, and environment variables, enabling agents to adapt behavior based on system capabilities and configuration. Queries OS-level APIs to gather hardware information (CPU, memory, display resolution) and software environment (installed packages, PATH, environment variables). Provides this metadata to agents for capability negotiation and conditional execution.

Implements error handling and recovery mechanisms for failed operations, including retry logic with exponential backoff, state validation after operations, and detailed error reporting. Validates that operations succeeded by comparing expected state (e.g., window focus, clipboard contents) with actual state, and provides detailed error messages including OS error codes and recovery suggestions. Enables agents to detect and recover from transient failures without explicit error handling logic.

Tracks performance metrics for each operation including execution time, latency, and resource usage, enabling agents and developers to identify bottlenecks and optimize workflows. Records timing information for screenshot capture, input operations, and window management, and exposes metrics through MCP interface. Implements low-overhead instrumentation that doesn't significantly impact operation latency.