OpenAgents

OpenAgents implements a service-oriented architecture that routes user requests to one of three specialized agent types (Data, Plugins, Web) based on task intent. The backend Flask server maintains a unified message flow interface while each agent type implements its own execution logic, with shared adapters handling stream parsing, memory callbacks, and data models. This modular design allows agents to be independently deployed and scaled while maintaining a consistent interface for the frontend.

The Data Agent provides a specialized toolkit for data manipulation, analysis, and visualization by executing Python and SQL code in a sandboxed environment. It integrates with the backend's memory system to maintain context across multiple data operations, supports file uploads (CSV, JSON, images), and generates visualizations through matplotlib/plotly. The agent uses LLM-guided code generation to translate natural language data requests into executable Python/SQL, with streaming output to provide real-time feedback during long-running computations.

OpenAgents maintains a registry of 200+ plugins with structured metadata (name, description, parameters, authentication requirements, category). Plugins are registered with JSON schemas describing their inputs/outputs, enabling the LLM to understand plugin capabilities and select appropriate plugins based on user intent. The registry supports plugin discovery, parameter validation, and authentication management, allowing new plugins to be added without modifying agent code.

The Data Agent generates executable Python and SQL code from natural language requests using the LLM, then executes the code in a sandboxed environment with access to uploaded data. The sandbox provides a controlled execution context with access to common data libraries (pandas, numpy, matplotlib, plotly) while isolating dangerous operations. Generated code is logged and can be reviewed before execution, providing transparency into what the agent is doing.

The Web Agent integrates vision-language models (GPT-4V, Claude Vision) to interpret screenshots of web pages and understand their visual layout, content, and interactive elements. The agent captures screenshots during browsing, sends them to the vision model with a task description, and receives natural language descriptions of page content and recommended actions. This enables the agent to interact with websites without relying on DOM parsing or explicit selectors, making it adaptable to varied website designs.

OpenAgents maintains a conversation history within each session that includes user messages, agent responses, and file references. The system allows agents to access previous messages and uploaded files throughout a conversation, enabling multi-turn interactions where agents build on prior context. File uploads are stored with metadata (filename, upload time, size) and can be referenced in subsequent requests without re-uploading, improving user experience for iterative analysis.

The Plugins Agent provides access to 200+ third-party APIs (shopping, weather, scientific tools, etc.) through a unified plugin registry system. The agent uses LLM-based reasoning to automatically select relevant plugins based on user intent, constructs appropriate API calls with parameter binding, and handles response parsing/formatting. Plugins are registered with metadata (description, parameters, authentication requirements) that the LLM uses for selection, enabling the agent to discover and invoke APIs without explicit user specification.

The Web Agent enables autonomous web browsing through a Chrome extension that allows the agent to navigate websites, extract information, and interact with web pages (clicking, form filling, scrolling). The agent receives visual feedback (screenshots) from the browser, uses vision-language models to understand page content, and generates browser commands (navigate, click, extract text) to accomplish user goals. This creates a closed-loop system where the agent observes page state, reasons about next actions, and executes them iteratively until the task completes.

OpenAgents implements a streaming architecture where agent responses are sent to the frontend in real-time via WebSocket connections rather than waiting for complete execution. The backend uses streaming callbacks and adapters to capture intermediate outputs (code execution results, API responses, reasoning steps) and forward them to the frontend as they occur. This enables users to see progress during long-running operations (data analysis, web scraping) without waiting for final results, improving perceived responsiveness and allowing early termination of slow operations.

OpenAgents provides a session-based memory system where conversation history, file uploads, and agent execution context are persisted in MongoDB and cached in Redis. The memory system is shared across all three agent types through common adapters, allowing agents to reference previous messages, uploaded files, and past analysis results within a session. The backend manages memory lifecycle (creation, updates, cleanup) and provides APIs for agents to read/write context, enabling multi-turn conversations where agents build on prior interactions.

OpenAgents abstracts LLM interactions through a provider-agnostic interface that supports multiple LLM backends (OpenAI, Anthropic, Ollama, etc.). The backend maintains LLM configuration (model selection, temperature, max tokens) and routes agent requests to the appropriate provider based on configuration. This allows users to switch LLM providers without changing agent code, and enables cost optimization by using different models for different tasks (e.g., cheaper models for simple tasks, GPT-4 for complex reasoning).

OpenAgents provides a web-based chat interface built with Next.js that allows users to select agents, upload files, and interact with agents through a conversational UI. The frontend manages application state (current agent, conversation history, uploaded files) and communicates with the backend via REST APIs and WebSocket connections. The interface includes file upload/download capabilities, agent selection dropdowns, and streaming message display, creating a unified entry point for all three agent types.

OpenAgents provides Docker containerization for both frontend and backend services, enabling consistent deployment across development, staging, and production environments. The deployment uses environment variables for configuration (API keys, LLM provider selection, database connections), allowing the same Docker images to be deployed with different configurations. Docker Compose orchestration is provided for local development, simplifying setup of the full stack (frontend, backend, MongoDB, Redis).

OpenAgents provides a framework for creating custom agents by extending base agent classes and implementing required methods (execute, parse_response, etc.). The framework defines a common interface that all agents must implement, allowing new agents to be added without modifying core backend logic. Custom agents inherit shared infrastructure (memory, callbacks, streaming adapters) automatically, reducing boilerplate and ensuring consistency with existing agents.