Agent-S

Agent-S uses Large Multimodal Models (LMMs) to observe desktop screenshots, extract visual and textual elements through grounding mechanisms, and generate coordinate-based GUI actions. The system maintains a unified LMM provider abstraction layer supporting OpenAI, Anthropic, and other LMM backends, with message management that preserves visual context across multi-turn interactions. Actions are grounded to screen coordinates via PyAutoGUI execution primitives, enabling pixel-perfect GUI automation.

Agent-S2 implements a two-level planning hierarchy where a Manager agent decomposes high-level tasks into subtasks using DAG-based planning, and Worker agents execute individual subtasks with focused context. The Manager maintains task dependencies and execution order, while Workers operate with reduced context windows, improving efficiency and enabling parallel execution. This architecture is implemented via manager_step() and worker_step() methods with shared knowledge base integration for state synchronization.

Agent-S3 integrates a local coding environment where agents can generate and execute Python code directly for programmatic operations. The CodeAgent component generates Python scripts for tasks like file I/O, data processing, or API calls, executing them in a controlled environment. Execution results are captured and fed back to the agent for further planning. This capability enables agents to choose between GUI automation and direct code execution based on task requirements, improving efficiency for programmatic tasks.

Agent-S uses PyAutoGUI as the unified execution backend for GUI automation across Linux, macOS, and Windows. The system abstracts platform-specific differences through a coordinate-based action interface, translating high-level action descriptions (click, type, scroll) into PyAutoGUI commands. Platform-specific implementations handle display scaling, coordinate system differences, and OS-specific input methods. This approach enables agents to control any GUI application without platform-specific rewrites.

Agent-S integrates RAG capabilities through embedding engines that encode task descriptions, procedural memory, and historical execution traces into vector space. The system retrieves relevant examples and procedures based on semantic similarity to the current task, augmenting the agent's context with relevant knowledge. This approach combines procedural memory with dynamic retrieval, enabling agents to leverage task-specific knowledge without explicit prompt engineering.

Agent-S integrates OCR services (Tesseract, EasyOCR, or cloud-based) to extract text from screenshots and localize UI elements. The OCR pipeline identifies text regions, extracts content, and maps text to screen coordinates, enabling agents to ground natural language references to specific UI elements. This capability is essential for text-based grounding when visual features alone are insufficient. OCR results are cached and reused across multiple agent steps to reduce latency.

Agent-S implements signal handling for graceful shutdown, allowing agents to save execution state, close resources, and terminate cleanly on interrupt signals (SIGINT, SIGTERM). The system preserves execution traces, screenshots, and agent state to enable resumption or post-mortem analysis. This capability is essential for long-running agents where interruption is expected and state recovery is important.

Agent-S3 simplifies the architecture to a single Worker agent with integrated CodeAgent capability, eliminating manager overhead while maintaining task completion accuracy. The agent can generate and execute Python code directly in a local coding environment for programmatic operations, bypassing GUI interactions when more efficient. This flat design uses a single predict() method with reflection-based error recovery, reducing latency and complexity compared to hierarchical versions.

Agent-S implements Behavior Best-of-N sampling where the agent generates multiple action trajectories (rollouts) in parallel, evaluates them using a scoring function, and selects the highest-scoring trajectory. This in-context reinforcement learning approach improves accuracy without retraining by leveraging the LMM's ability to reason about action quality. The system supports configurable rollout counts (typically 3-5) and can be combined with reflection mechanisms for iterative refinement.

Agent-S defines a unified Agent-Computer Interface abstraction that standardizes how agents perceive and interact with computers. The ACI layer implements visual grounding (mapping LMM-generated descriptions to screen coordinates) and text grounding (extracting and localizing UI text elements). OSWorldACI is the primary implementation, using OCR services and coordinate systems to translate high-level action descriptions into pixel-precise PyAutoGUI commands. The system supports multiple coordinate systems and platform-specific implementations.

Agent-S maintains procedural memory through structured prompt templates and in-context examples that guide agent behavior. The system stores successful action sequences, error recovery patterns, and task-specific procedures as reusable prompts. Memory is managed through a prompt registry that can be dynamically loaded based on task context, enabling agents to leverage past experiences without explicit fine-tuning. This approach combines static procedural knowledge with dynamic context selection.

Agent-S1 implements GraphSearchAgent using graph-based planning where the agent explores a search tree of possible action sequences, maintaining state nodes and evaluating paths to the goal. The system uses hierarchical exploration with expand() and predict() methods to grow the search tree, pruning low-probability branches. This approach combines classical planning (graph search) with LMM-based heuristics for node evaluation, enabling systematic exploration of action spaces.

Agent-S provides a unified LMM provider abstraction layer that normalizes interfaces across OpenAI, Anthropic, and other LMM backends. The system manages message history with vision context preservation, handling image encoding, token counting, and provider-specific API differences transparently. LMMAgent base class implements message management with support for multi-turn conversations, image attachment, and context window optimization. This abstraction enables swapping LMM providers without changing agent logic.

Agent-S includes native integration with OSWorld and WindowsAgentArena evaluation frameworks, providing standardized task definitions, environment setup, and result evaluation. The system implements evaluation scripts that run agents against benchmark tasks, collect execution traces, and compute accuracy metrics. Integration includes parallel evaluation support for Azure deployment, enabling large-scale benchmark runs. Evaluation results are processed and compared against baseline performance.

Agent-S implements reflection mechanisms where agents analyze failed actions, identify error causes, and generate corrective actions. The system uses LMM reasoning to understand why an action failed (e.g., 'button not found', 'incorrect input format') and generates alternative approaches. Reflection can be applied iteratively, building a history of failed attempts and lessons learned. This approach enables agents to recover from transient failures and adapt to unexpected UI changes.