Notte

Notte's LLM engine abstracts multiple LLM providers (OpenAI, Anthropic, Gemini, Ollama) through a unified interface that handles provider-specific API differences, token counting, and context window management. The engine integrates with the agent system to enable reasoning loops where agents analyze DOM state, decide on actions, and iterate until task completion. This architecture decouples agent logic from LLM provider selection, allowing runtime switching between models without code changes.

Notte manages browser sessions as first-class objects that maintain DOM state, navigation history, and interaction context across multiple agent steps. Sessions can execute locally (via Playwright/Puppeteer) or remotely (via Notte's cloud API), with the same SDK interface abstracting the execution location. The session layer handles browser lifecycle (launch, navigate, close), screenshot capture, and DOM observation, feeding state back to agents for decision-making.

Notte integrates documentation systems and knowledge bases into agent context, enabling agents to reference documentation, FAQs, and domain knowledge during reasoning. The system can ingest documentation from multiple sources (websites, PDFs, APIs) and provide agents with relevant context based on task description. This reduces hallucination and improves agent accuracy by grounding reasoning in authoritative sources.

Notte provides comprehensive observability through execution traces (step-by-step logs of agent reasoning and actions), detailed logs (browser events, API calls, errors), and replay functionality (re-execute workflows with recorded state). The system captures DOM snapshots at each step, enabling developers to inspect what the agent saw and why it made decisions. Traces can be exported for analysis, debugging, and compliance auditing.

Notte supports batch processing of multiple URLs or tasks through a single workflow, with structured data extraction and output validation. The system can extract data from multiple pages, validate extracted data against schemas, and combine results into a single output. Extraction rules can be defined declaratively (CSS selectors, XPath, LLM-based extraction), and results are validated before returning to ensure consistency and correctness.

Notte converts browser DOM into a structured, accessibility-aware representation that agents can reason about without parsing raw HTML. The system builds an observation model that includes element IDs, text content, ARIA labels, and interactive properties, enabling agents to target elements by semantic meaning rather than CSS selectors. This abstraction layer sits between the browser controller and agent reasoning, providing a normalized view of page state regardless of underlying HTML structure.

Notte's action system provides a structured interface for browser interactions, supporting both deterministic scripts (click, type, navigate) and AI-driven actions where agents decide what to do based on page state. Actions are validated, logged, and executed through a unified controller that abstracts browser implementation details. The system enables mixing scripted workflows (for known steps) with agent-driven exploration (for variable paths), allowing hybrid automation strategies.

Notte provides a vault system for securely storing and injecting credentials (API keys, passwords, auth tokens) into browser sessions without exposing them in code or logs. The vault integrates with agent execution, allowing agents to request credentials for specific services (e.g., 'login to Gmail') without knowing the actual credentials. Personas can be defined with associated credentials, enabling agents to act as different users or service accounts.

Notte's workflow system enables defining multi-step automation sequences where each step can be a scripted action, an agent decision, or a conditional branch. Workflows maintain state across steps (variables, extracted data, session context), allowing later steps to reference earlier results. The system supports parallel execution, error handling, and rollback, providing a declarative way to compose complex automation tasks from simpler building blocks.

Notte integrates with Model Context Protocol (MCP) servers, enabling agents to call external tools and access data sources (APIs, databases, knowledge bases) during browser automation. The integration allows agents to reason about when to use external tools (e.g., 'fetch weather data from API') and incorporate results into browser interactions. MCP servers are registered with the agent, providing a standardized interface for tool discovery and execution.

Notte provides validation and sandboxing for agent scripts and custom code, preventing malicious or buggy code from compromising browser sessions or accessing unauthorized resources. The system validates script syntax, checks for dangerous operations (file system access, network calls outside allowed domains), and executes scripts in a restricted environment. Security policies can be defined per agent or workflow, controlling what resources agents can access.

Notte supports both cloud-hosted agent execution (via Notte API) and local execution (via open-source core), with a unified SDK that abstracts the execution location. Agents can be deployed to the cloud for scalability and reliability, or run locally for development and privacy. The system handles agent serialization, remote execution, and result streaming, enabling seamless switching between local and cloud backends.

Notte optimizes performance through multiple strategies: caching DOM observations to reduce re-parsing, batching multiple actions into single browser commands, and executing multiple sessions in parallel for independent tasks. The system profiles agent execution (token usage, action latency, reasoning time) and provides metrics for optimization. Developers can configure caching policies, batch sizes, and parallelism levels based on workload characteristics.