Peekaboo

Captures screenshots using ScreenCaptureKit (macOS 12.3+) with automatic CGWindow fallback, supporting Retina scaling (2x on HiDPI displays), multi-display targeting via screen index, window-scoped capture by app name/PID/window ID, and menu bar capture including status bar extras. The capture engine is abstraction-layered to allow runtime selection between ScreenCaptureKit and legacy CGWindow APIs based on availability and performance characteristics.

Detects interactive UI elements (buttons, text fields, menus, etc.) using macOS Accessibility APIs (AXUIElement) with fallback to vision-based element detection when accessibility metadata is unavailable. The system maintains a semantic element registry that maps detected elements to their accessibility attributes (role, label, value, enabled state) and enables deterministic interaction via native accessibility actions (click, type, select) rather than pixel-based mouse movement.

Manages storage of element detection snapshots (visual crops of detected UI elements) on disk with automatic cleanup to prevent unbounded storage growth. The system stores snapshots in a configurable directory, tracks snapshot metadata (timestamp, element ID, size), and implements cleanup policies (age-based, size-based, LRU). Snapshots are used by vision models to analyze specific UI elements without re-capturing the entire screen.

Provides a native macOS application (Peekaboo.app) that runs in the status bar and offers a visual inspector for debugging Peekaboo operations. The app displays real-time screenshots, detected UI elements, and execution logs; allows users to manually trigger captures and interactions; and provides a settings interface for configuration. The app maintains a persistent connection to the Peekaboo service and streams events in real-time.

Integrates macOS native speech recognition (via Speech framework) to enable voice-based interaction with the Peekaboo agent. The system captures audio input, transcribes it to text using on-device speech recognition, and passes the transcribed text to the agent as a natural language instruction. Speech recognition runs asynchronously and supports real-time transcription feedback.

Implements a comprehensive error handling system that captures detailed diagnostic information (stack traces, system state, screenshots) when operations fail, provides human-readable error messages, and implements recovery strategies (retry with backoff, fallback paths, state rollback). The system categorizes errors by severity and type, enabling targeted recovery logic and diagnostic reporting.

Executes deterministic UI interactions (click, type, select, scroll, drag) using native macOS accessibility actions (AXPress, AXSetValue, etc.) when elements expose accessibility metadata, with fallback to synthetic input (CGEvent-based mouse/keyboard events) for inaccessible elements. The system maintains an interaction queue that serializes actions to prevent race conditions and includes error recovery logic that retries failed interactions with exponential backoff.

Manages macOS window lifecycle and space (virtual desktop) navigation using a heuristic-based window selection system that ranks windows by relevance (foreground status, recent focus, window type). The system can enumerate all windows, filter by application, activate windows, move windows between spaces, and handle window-scoped operations. Window selection heuristics account for hidden windows, minimized windows, and multiple windows from the same application.

Controls application lifecycle (launch, quit, force quit) and automates menu bar and dock interactions using a combination of accessibility APIs and synthetic input. The system can launch applications by bundle identifier or path, terminate applications gracefully or forcefully, interact with menu items (click, check state), and manipulate dock icons. Menu automation includes special handling for hierarchical menus and dynamic menu items that appear/disappear based on application state.

Exposes Peekaboo capabilities as an MCP (Model Context Protocol) server that can be connected to AI clients (Claude Desktop, Cursor, other MCP-compatible tools) via stdio or HTTP transport. The MCP server implements a tool registry that maps Peekaboo commands to MCP tool definitions with JSON schema validation, handles async tool execution with streaming output, and manages session state across multiple tool calls. The server supports both synchronous and asynchronous tool execution patterns.

Implements a complete AI agent execution pipeline that orchestrates tool calls, manages session state, and supports multiple LLM models via a model selection system (Tachikoma). The pipeline includes tool registration, JSON schema validation, execution queuing, error handling with retry logic, and event streaming for real-time progress updates. The system supports both interactive chat mode (multi-turn conversation) and batch execution mode (single-shot automation).

Provides a hierarchical configuration system that manages Peekaboo settings, LLM credentials, and macOS permissions state. The system supports configuration files (YAML/JSON), environment variables, and credential storage via secure macOS Keychain integration. The permissions tracking system monitors which macOS permissions have been granted (Screen Recording, Accessibility, etc.) and provides diagnostic information when permissions are missing.

Provides a comprehensive CLI interface (peekaboo binary) that exposes all Peekaboo capabilities as subcommands organized by category: vision commands (screenshot, detect-elements), interaction commands (click, type, select), window management commands (list-windows, activate-window), application control commands (launch, quit), menu automation commands (click-menu), and system utility commands. Each command supports JSON output for programmatic parsing and includes detailed help text.

Implements an interactive chat interface that enables multi-turn conversations with an AI agent, maintaining session state across multiple user inputs. The system manages conversation history, tool execution context, and screenshot state, allowing users to give natural language instructions and receive feedback in real-time. The chat mode supports both CLI-based interaction and programmatic session management via the Agent API.