LiteWebAgent

Processes web pages by combining accessibility tree (axtree) extraction, DOM element parsing, and screenshot analysis to build a unified representation of page structure and content. The system extracts interactive elements, their positions, and semantic relationships, enabling VLMs to reason about page layout without raw HTML. This multi-modal approach allows agents to understand both the logical structure (via axtree) and visual presentation (via screenshots) simultaneously.

Converts high-level natural language instructions into executable multi-step action sequences using specialized planning agents (HighLevelPlanningAgent, ContextAwarePlanningAgent). The system decomposes complex goals into sub-tasks, reasons about dependencies, and generates structured action plans that can be executed by function-calling agents. Planning agents leverage VLM reasoning to understand task semantics and generate contextually appropriate action sequences.

Integrates multiple Vision-Language Model providers (OpenAI GPT-4V, Anthropic Claude, etc.) through a unified interface, handling model-specific API differences, function-calling schemas, and response formats. The system abstracts away provider-specific details, allowing agents to work with different VLMs without code changes. Configuration specifies the model provider and parameters, enabling easy model switching.

Provides browser automation capabilities through integration with Playwright and Selenium, handling browser lifecycle management, page navigation, element interaction, and screenshot capture. The system abstracts browser-specific details, providing a unified interface for common automation tasks (click, type, scroll, submit). Async support enables non-blocking browser operations for concurrent agent execution.

Tracks agent execution state throughout a workflow, capturing action sequences, page states, and outcomes at each step. The system maintains a complete execution trace that can be replayed, analyzed, or used for debugging. State management handles browser session state, agent memory state, and workflow progress, enabling recovery from failures and analysis of execution paths.

Executes web interactions through a structured function-calling interface where web actions (click, type, scroll, submit) are registered as callable functions with defined schemas. The FunctionCallingAgent maps VLM-generated function calls to actual browser automation commands, handling parameter validation and execution. This approach decouples action planning from execution, enabling tool reuse across different agent types and VLM providers.

Stores and retrieves past web automation workflows to inform future agent decisions through the Agent Workflow Memory (AWM) module. The system captures execution traces (states, actions, outcomes) and enables context-aware agents to retrieve relevant past workflows, learning from successes and failures. This memory integration allows agents to adapt behavior based on historical context without explicit fine-tuning.

Implements Set-of-Mark (SoM) technique where interactive elements on a webpage are visually marked with unique identifiers (numbers, labels) in a modified screenshot, and agents interact with elements by referencing these marks in natural language prompts. The PromptAgent uses this visual marking approach to ground agent instructions in specific UI elements without requiring precise coordinate calculations or DOM element selection.

Exposes agent capabilities through multiple user interfaces: command-line interface for scripting, web playground for interactive testing, Chrome extension for in-browser automation, and Python API for programmatic integration. Each interface connects to a shared FastAPI backend that manages agent lifecycle, state, and execution. This multi-interface design allows different user personas (developers, non-technical users, end-users) to interact with the same underlying agent system.

Implements a FastAPI server that manages agent lifecycle, handles concurrent requests, and provides async execution of web automation tasks. The backend uses async/await patterns to enable non-blocking agent execution, allowing multiple agents to run concurrently without blocking the server. State management is handled through async API services that coordinate browser sessions, memory access, and result collection.

Implements a factory pattern (agent_factory.py) that centralizes agent instantiation and allows developers to select from multiple agent types (FunctionCallingAgent, PromptAgent, HighLevelPlanningAgent, ContextAwarePlanningAgent) through a unified interface. The factory handles model configuration, tool registry setup, and memory initialization, abstracting away the complexity of agent construction. This pattern enables easy switching between agent types without changing client code.

Provides an evaluation suite that benchmarks agent performance against WebArena and X-WebArena datasets, which contain realistic web automation tasks with ground-truth solutions. The framework measures success rates, action efficiency, and other metrics to quantify agent performance. This enables systematic comparison of different agent types, models, and strategies on standardized benchmarks.

Extracts interactive elements (buttons, links, input fields, etc.) from web pages and maps them to precise coordinates and DOM selectors. The system identifies clickable regions, input targets, and form elements, providing agents with a structured list of available interactions. Coordinate mapping enables accurate element targeting for browser automation, while DOM selectors provide fallback targeting methods.