open-chatgpt-atlas

Captures full-page screenshots, sends them to Google's Gemini 2.5 Computer Use model for visual understanding, and receives normalized 1000x1000 coordinate grids for precise click, type, and scroll actions. This approach enables the AI to interact with any web UI without requiring DOM parsing or element selectors, making it resilient to dynamic content and obfuscated interfaces.

Routes natural language requests through Composio's Tool Router to generate direct API calls against 500+ integrated services (Gmail, Slack, GitHub, Salesforce, etc.) instead of simulating UI clicks. The system maintains a schema registry of available tools, matches user intent to applicable APIs, and executes calls with proper authentication and error handling, bypassing visual automation entirely for supported platforms.

Routes requests to different LLM models based on task type: Gemini 2.5 Computer Use for visual browser automation, standard Gemini for text-based tool selection and reasoning, and Composio's Tool Router for API-based execution. Implements fallback logic to switch models if the primary choice fails or times out.

Captures full-page screenshots from the browser viewport, normalizes them to a 1000x1000 coordinate grid regardless of actual screen resolution or DPI, and sends them to the vision model. This normalization ensures that coordinate predictions from the model are consistent across different devices and screen sizes, with a reverse-mapping step to translate normalized coordinates back to actual pixel positions.

Implements automatic retry logic for transient failures (API timeouts, rate limits, network errors) using exponential backoff with jitter. Failed actions are logged with full context (screenshot, prompt, error message) for debugging, and the agent can decide whether to retry the same action, try an alternative approach, or escalate to the user.

Shares a unified core logic layer across two distinct deployment targets: a Manifest V3 Chrome Extension (using chrome.debugger and content script injection for tab automation) and a standalone Electron desktop app (using BrowserView and native IPC for full browser control). Both targets implement the same AI routing logic but use different automation primitives and persistence mechanisms (chrome.storage.local vs electron-store).

All API requests to model providers (Google Gemini, Composio) are made directly from the client (extension or desktop app) without routing through an intermediary backend server. This eliminates the need for a centralized proxy, reduces latency, and ensures user prompts and browser state never touch a third-party server beyond the official API providers.

Implements a multi-turn agentic loop where the LLM receives tool availability (both Computer Use and Tool Router), decides which tool to invoke, executes the action, observes the result (screenshot or API response), and iteratively refines its approach. The system handles streaming responses from the LLM, allowing real-time display of reasoning and action execution without waiting for full completion.

The Chrome Extension uses content scripts injected into the active tab to interact with the DOM, capture user interactions, and relay information back to the background service worker. This enables the extension to read page structure, inject JavaScript, and monitor network activity without requiring full debugger protocol access for every interaction.

The Chrome Extension's background service worker acts as the orchestration hub, receiving messages from content scripts and the side panel UI, routing them to the appropriate handler (Computer Use or Tool Router), and managing the agentic loop lifecycle. Uses Manifest V3's message passing API to coordinate between extension components while respecting the 5-minute execution timeout.

The Chrome Extension's side panel provides a chat-like interface where users input natural language prompts and observe real-time agent execution. The UI displays streaming LLM responses, screenshots with action annotations, and tool execution results, allowing users to monitor and interrupt the agent mid-execution.

The Electron desktop app uses Inter-Process Communication (IPC) to separate the main process (which controls BrowserView and system APIs) from the renderer process (which hosts the UI). The main process handles browser automation and API calls, while the renderer displays results. This isolation provides security (renderer cannot directly access system APIs) and stability (renderer crashes don't crash the main process).

Manages user settings (API keys, model preferences, automation mode selection) with different persistence backends for Chrome Extension (chrome.storage.local with 10MB quota) and Electron (electron-store with filesystem-based JSON). Settings are loaded at startup and can be modified via a dedicated settings page, with validation and encryption for sensitive credentials.