browser-use

Translates LLM decisions into browser actions by maintaining a bidirectional bridge between language model outputs and Chrome DevTools Protocol (CDP) commands. The Agent system executes a loop where it captures browser state (DOM, screenshots, page metadata), sends structured context to an LLM provider (OpenAI, Anthropic, Gemini, or local models), parses the LLM's action schema output, and executes actions like click, type, navigate, and extract through CDP. Includes built-in error recovery, loop detection, and behavioral nudges to prevent agent stalling.

Converts raw HTML/CSS/JavaScript DOM trees into LLM-readable markdown and text formats by traversing the DOM, detecting interactive elements (buttons, inputs, links), calculating visibility based on CSS and viewport geometry, and assigning stable numeric indices. The DOM Processing Engine uses a Watchdog pattern to monitor DOM mutations, re-serialize only changed subtrees, and maintain coordinate mappings for accurate click targeting. Outputs include markdown extraction (headings, text content), HTML serialization with element indices, and a browser state summary with page title and URL.

Extracts structured data from web pages by defining a schema (JSON Schema or Pydantic model) and using the agent to navigate to the relevant page, locate the data, and extract it in the specified format. The extraction action validates the extracted data against the schema and returns structured output (JSON, Python objects). Supports both single-page extraction (extract data from current page) and multi-page extraction (navigate through pages and aggregate results). Includes error handling for schema validation failures and retry logic for incomplete extractions.

Tracks agent execution metrics (actions taken, LLM calls, tokens used, time elapsed) and estimates costs based on LLM provider pricing. Collects telemetry data on agent performance, error rates, and task completion rates. Supports optional cloud sync to aggregate metrics across multiple agent runs and deployments. Provides detailed cost breakdowns per LLM provider and per task. Includes privacy controls to disable telemetry collection if needed.

Enables developers to define custom actions beyond the built-in set (click, type, navigate, extract) by registering custom tool classes that implement a standard interface. Custom tools are integrated into the action execution pipeline and exposed to the LLM as available actions. Supports tool-specific error handling, validation, and documentation. Tools are discovered at runtime and can be dynamically registered or unregistered. Includes examples and templates for common custom tools (screenshot, download, execute JavaScript).

Abstracts LLM provider differences (OpenAI, Anthropic Claude, Google Gemini, local Ollama) behind a unified interface that automatically optimizes action schemas per provider's capabilities. Handles provider-specific structured output formats (OpenAI's JSON mode, Anthropic's tool_use, Gemini's function calling), manages token counting and cost tracking, implements exponential backoff retry logic for rate limits and transient failures, and serializes agent state into provider-specific message formats. Supports both cloud-based and local LLM backends with fallback chains.

Detects when an agent enters repetitive action cycles (e.g., clicking the same button repeatedly, typing the same text) by comparing recent action history and DOM snapshots. When a loop is detected, the system applies behavioral nudges: suggesting alternative actions, modifying the system prompt to encourage exploration, or triggering a 'judge' evaluation to assess task progress. Uses heuristics like action frequency analysis, DOM change detection, and coordinate repetition to identify stalls. Includes configurable thresholds and nudge strategies.

Automatically compresses agent conversation history to fit within LLM context windows by summarizing old messages, removing redundant state information, and prioritizing recent actions. Uses a compaction strategy that identifies the most important historical context (e.g., task definition, key decisions) while discarding verbose intermediate steps. Tracks token usage across the conversation and triggers compaction when approaching the LLM's max_tokens limit. Maintains a compact representation of agent state (current page, recent actions, key findings) to preserve context fidelity.

Manages Chrome browser instances through a SessionManager that handles process lifecycle (launch, shutdown, graceful termination), maintains a pool of CDP connections for multi-tab scenarios, and persists browser state (cookies, localStorage, sessionStorage) across sessions via storage state JSON files. Supports browser profile configuration (user data directory, launch arguments, proxy settings) and handles popup/dialog interactions. Implements signal handling for graceful shutdown and cleanup of browser processes on agent termination.

Monitors DOM mutations in real-time using a Watchdog pattern that listens for browser events (DOMContentLoaded, load, mutation events) and triggers re-serialization only when the DOM changes. Maintains a cache of the last serialized DOM state and compares new snapshots to detect meaningful changes. Supports event filtering to ignore cosmetic changes (e.g., CSS animations) and focus on structural changes (e.g., new elements, attribute changes). Enables efficient state tracking without full-page re-parsing on every step.

Executes LLM-generated actions (click, type, navigate, extract, scroll, wait) through a unified pipeline that validates action schemas, translates them to CDP commands, handles execution errors, and implements exponential backoff retry logic. Supports action-specific error handling (e.g., element not found, stale element reference) with recovery strategies like re-serializing the DOM and retrying. Tracks action execution state and provides detailed error traces for debugging. Includes built-in actions for common tasks (click, type, navigate, extract) and extensibility for custom actions.

Evaluates agent progress on a task by analyzing the execution trace (sequence of actions, state changes, LLM decisions) and determining if the agent is making meaningful progress toward the goal. The Judge uses an LLM to assess whether recent actions are productive, whether the agent has achieved the task objective, or whether it should try a different approach. Provides structured feedback on task completion status, confidence scores, and suggestions for next steps. Integrates with loop detection to trigger evaluation when the agent may be stuck.

Provides multiple interfaces for running browser-use agents: a Python API for programmatic integration, a command-line interface (CLI) for one-off tasks, a text-based user interface (TUI) using Textual for interactive debugging, and a Model Context Protocol (MCP) server for integration with other AI tools. Each interface abstracts the underlying agent logic while providing interface-specific features (e.g., TUI shows live screenshots and action logs, MCP server exposes agent capabilities as tools). Enables seamless switching between development, testing, and production deployment modes.