bytebot

Executes multi-step desktop automation tasks from natural language descriptions by implementing an observe-act-verify cycle where the AgentProcessor polls the desktop state via screenshot, sends observations to an LLM (OpenAI, Anthropic, or Gemini), receives computer actions, executes them through the ComputerUseService, and repeats until task completion. The system maintains full task state in PostgreSQL and broadcasts real-time progress through WebSocket events, enabling both autonomous execution and human intervention via takeover mode.

Abstracts LLM provider differences through a unified interface that supports OpenAI, Anthropic, and Google Gemini with native support for their computer-use/vision APIs. The AgentProcessor routes task execution to the configured LLM provider, handles provider-specific function calling schemas, manages token context windows, and implements fallback logic. Each provider integration handles vision input (desktop screenshots), tool/function definitions for computer actions, and streaming response parsing.

Supports password manager integration (e.g., KeePass, 1Password) to automatically fill authentication credentials during task execution. The agent can request credentials from the password manager, which are injected into login forms without exposing them in task logs or agent messages. This enables secure automation of workflows requiring authentication without hardcoding credentials.

Maintains a complete message history for each task, including agent reasoning, tool calls, observations, and user messages. Messages are stored in PostgreSQL with different content types (text, images, tool calls, results) and displayed in the web UI in chronological order. This provides full transparency into the agent's decision-making process and enables debugging of failed tasks.

Supports basic task scheduling where tasks can be configured to run at specific times or on a recurring basis. The AgentScheduler manages task scheduling logic, persisting schedule configurations to PostgreSQL and triggering task execution at scheduled times. This enables automation of routine workflows without manual intervention.

Provides an isolated, containerized Ubuntu desktop environment running inside Docker where all desktop automation occurs. The bytebotd NestJS daemon (port 9990) exposes the desktop through a noVNC web client for real-time visual monitoring, handles VNC input tracking to detect human intervention, and manages the lifecycle of desktop applications. The environment includes pre-configured tools (browser, terminal, file manager) and supports password manager integration for authentication flows.

Manages the complete lifecycle of automation tasks (creation, queuing, execution, completion, failure) through the TasksService API and TasksGateway WebSocket broadcaster. Tasks are persisted to PostgreSQL with state transitions (pending → running → completed/failed), and all state changes are broadcast in real-time to connected clients via WebSocket events. The system supports task scheduling, file attachment handling, and message history tracking with different content types (text, images, tool calls).

Enables users to upload files (PDFs, spreadsheets, documents) which are stored and injected into the LLM context during task execution. The system handles file parsing, storage in PostgreSQL (via Prisma), and inclusion in agent messages as base64-encoded content or extracted text. This allows the agent to process documents without downloading them from external sources, reducing task complexity and improving privacy.

Executes low-level desktop automation actions (mouse clicks, keyboard input, file operations, screenshot capture) through the ComputerUseService running in the bytebotd daemon. Actions are received as structured JSON commands from the LLM, validated, and executed against the Ubuntu desktop environment. The system tracks action history, handles action failures gracefully, and provides feedback to the agent for the next observation cycle.

Detects human intervention during task execution by monitoring VNC input events and allows seamless takeover where a human operator can control the desktop while the agent pauses. The system tracks input sources (agent vs. human), maintains task state during takeover, and enables the agent to resume execution after human actions. This is implemented through VNC input event polling and task state management in the TasksService.

Exposes the bytebotd desktop service as an MCP (Model Context Protocol) endpoint, allowing external LLM clients and tools to invoke computer actions directly. The MCP integration provides a standardized interface for tool definition and invocation, enabling Bytebot to be used as a backend for other AI systems or integrated into larger MCP-based workflows. This is implemented through an MCP server running in the bytebotd daemon that translates MCP tool calls to ComputerUseService actions.

Exposes a comprehensive REST API (on port 9991) and WebSocket API for programmatic task creation, monitoring, and control. The REST API provides endpoints for task CRUD operations, file uploads, and computer action execution, while the WebSocket API enables real-time event streaming (task status changes, agent messages, desktop updates). This allows external systems and custom frontends to integrate with Bytebot without using the built-in web UI.

Provides a Next.js 15 web UI (bytebot-ui service on port 9992) with React components for task creation, task list management, task detail viewing, and real-time desktop visualization. The frontend proxies all backend communication (HTTP, WebSocket, VNC) through a custom Express server, eliminating CORS issues and enabling seamless integration. The desktop viewer displays live VNC stream and agent messages, while the task interface supports file uploads and parameter configuration.