web-agent-protocol

Records user interactions (clicks, typing, navigation) in a live browser session by instrumenting Playwright's event listeners and capturing DOM snapshots at each interaction point. Stores interaction sequences with full DOM state, element selectors, and coordinate data to enable deterministic replay and agent learning from human demonstrations.

Replays recorded interaction sequences by resolving stored selectors (CSS, XPath, or coordinate-based) against the current DOM and executing the corresponding Playwright actions (click, type, navigate). Handles selector drift by falling back to alternative selector strategies and validates element visibility/interactability before execution.

Provides built-in assertions for validating interaction outcomes: element visibility, text content matching, URL changes, network request completion. Supports both immediate assertions (after each interaction) and deferred assertions (after workflow completion), enabling agents to verify that interactions succeeded and pages reached expected states.

Exposes recording and replay capabilities as MCP (Model Context Protocol) tools that LLM agents can invoke through a standardized interface. Implements MCP server protocol with tool definitions for start-recording, stop-recording, and replay-interaction, allowing Claude, other LLMs, and agent frameworks to orchestrate browser automation without direct library imports.

Selects elements for interaction using a cascading strategy: first attempts CSS selectors, falls back to XPath expressions, then uses coordinate-based selection as last resort. Validates element interactability (visibility, clickability) before returning and caches selector strategies that work for future reference, enabling robust element targeting across dynamic UIs.

Chains multiple recorded or programmatic interactions into a single executable workflow by composing interaction objects with dependency tracking and state validation between steps. Supports conditional branching based on page state (e.g., 'if element exists, click it; otherwise navigate') and error recovery strategies (retry with backoff, alternative action path).

Captures full DOM snapshots at interaction points and computes diffs between consecutive states to identify what changed (new elements, removed elements, attribute changes, text content changes). Provides structured representation of page state changes that agents can reason about, enabling learning from state transitions rather than just action sequences.

Manages Playwright browser instances, pages, and contexts with automatic lifecycle handling (launch, create page, close on error). Supports context isolation for parallel recording sessions and provides utilities for managing browser state (cookies, local storage, authentication) across interactions, enabling reproducible automation with consistent browser environment.

Converts recorded interaction sequences into training examples for LLM agents by pairing interaction contexts (page state, user goal) with executed actions. Generates structured prompts that teach agents to recognize similar situations and execute appropriate interactions, supporting few-shot learning where agents learn from 1-5 demonstrations before generalizing to new tasks.

Accepts natural language task descriptions (e.g., 'log in with email and password, then navigate to settings') and translates them into executable interaction sequences using LLM reasoning. The system decomposes goals into sub-tasks, selects appropriate recorded interactions or generates new ones, and executes them with error handling and goal validation.

Records interactions in a browser-agnostic format (semantic selectors, coordinate-independent actions) that can be replayed across different browsers (Chromium, Firefox, WebKit) without modification. Abstracts browser-specific APIs and handles rendering differences, enabling recorded workflows to work consistently regardless of browser engine.