AgentBench

Evaluates LLM agents across 8 heterogeneous task environments (OS, DB, KG, DCG, LTP, HH, WS, WB) through a unified Task interface that abstracts environment-specific implementations. Each task environment implements standard methods for sample retrieval, execution, and metric calculation, enabling systematic comparison of agent performance across fundamentally different domains without requiring agents to understand environment-specific APIs.

Manages bidirectional communication between agents and task environments through a Session abstraction that handles message exchange, conversation history tracking, and state management across multi-turn interactions. The Session interface standardizes how agents send actions and receive observations, enabling any agent implementation (LLM-based, rule-based, or hybrid) to interact with any task environment without environment-specific integration code.

Provides a Web Browsing environment (based on Mind2Web) that enables agents to navigate real websites and complete web-based tasks through simulated browser interactions. Agents can search, click links, fill forms, and extract information from web pages. The environment includes rendering of actual web pages and tracking of agent navigation paths. This environment tests agent capabilities in web understanding, navigation planning, and information extraction from complex web interfaces.

Provides an Operating System environment where agents interact with a Linux shell to execute commands, navigate file systems, and complete system administration tasks. Agents generate bash commands that are executed in a sandboxed Linux environment, with output returned as observations. The environment enforces resource limits and safety constraints to prevent harmful operations. This environment tests agent capabilities in command-line reasoning, file system navigation, and system administration.

Provides Database and Knowledge Graph environments where agents execute SQL queries or SPARQL queries against structured data. The DB environment includes a relational database with schema information; agents must formulate correct SQL queries to retrieve information. The KG environment includes a knowledge graph; agents must reason over relationships and formulate queries. Both environments test agent capabilities in structured data understanding, query formulation, and logical reasoning.

Provides a Household environment (based on ALFWorld) where agents complete household tasks in a simulated home environment. Tasks include finding objects, manipulating items, and completing household chores. The environment includes a 3D home simulation with object locations, agent actions (move, pick up, put down), and task success criteria. This environment tests agent capabilities in spatial reasoning, object tracking, and sequential task planning in realistic household scenarios.

Provides a Lateral Thinking Puzzles environment where agents solve puzzles that require non-obvious reasoning and constraint satisfaction. Puzzles present a scenario and agents must ask yes/no questions to determine the solution. The environment tracks questions asked, answers provided, and whether agents arrive at correct solutions. This environment tests agent capabilities in hypothesis formation, information seeking, and constraint-based reasoning.

Provides a Digital Card Game environment where agents play strategic card games requiring decision-making, resource management, and opponent modeling. The environment includes game rules, card mechanics, and win conditions. Agents must make strategic decisions about card play, resource allocation, and opponent prediction. This environment tests agent capabilities in strategic reasoning, game-theoretic thinking, and decision-making under uncertainty.

Orchestrates parallel evaluation of agents across task samples using a Task Controller that manages a pool of Task Workers and a Task Assigner for load distribution. The framework spawns worker processes to execute agent-task interactions in parallel, with the Task Assigner distributing samples across workers and the Task Controller aggregating results and computing final metrics. This architecture enables efficient benchmarking of multiple agents or multiple samples without sequential bottlenecks.

Computes task-specific evaluation metrics for each of the 8 environments through environment-specific metric implementations that understand domain semantics (e.g., success rate for OS tasks, SQL correctness for DB tasks, game score for DCG). The Task interface includes a metrics() method that each environment implements to calculate performance scores from agent interaction traces, enabling meaningful evaluation of agent behavior within each domain's context.

Provides an extension mechanism for adding custom task environments to the benchmark through documented Task and Agent interfaces that developers implement in Python. The framework includes extension guides (Extension_en.md, Extension_cn.md) that specify how to subclass Task base class, implement required methods (get_indices, execute, metrics), and integrate custom environments into the evaluation pipeline. This enables researchers to add domain-specific tasks beyond the 8 built-in environments.

Provides pre-built LLM agent implementations that interact with task environments through standardized Agent interface, supporting multiple LLM providers (OpenAI, Anthropic, and compatible endpoints). Agents implement decision-making logic that processes task observations and generates actions, with the framework handling API calls, token management, and response parsing. Agents can be configured with different LLM models and parameters without code changes.

Provides simple rule-based and heuristic agent implementations that serve as baselines for comparing LLM agent performance. These agents use pattern matching, keyword detection, or simple decision trees to generate actions without calling LLM APIs, enabling researchers to establish performance floors and understand how much value LLMs add over simple baselines. Naive agents implement the same Agent interface as LLM agents.

Enables declarative configuration of tasks, agents, and evaluation assignments through YAML/JSON configuration files rather than code. Configuration specifies task type, agent model, hyperparameters, sample selection, and worker allocation without requiring code changes. The framework parses configurations and instantiates appropriate task and agent implementations, enabling non-developers to run benchmarks and researchers to version control experimental setups.

Implements a complete Avalon card game environment (a social deduction game) with game engine, rule enforcement, and multi-agent gameplay. Agents play as knights or spies in a 5-player game requiring deception, negotiation, and strategic reasoning. The environment tracks game state, enforces rules, and computes win/loss metrics. This environment tests agent capabilities in social reasoning, deception detection, and strategic planning beyond simple task completion.

Provides a Web Shopping environment (based on WebShop) that simulates e-commerce interactions where agents must search for products, navigate product pages, add items to cart, and complete purchases. The environment includes a simulated product catalog, search functionality, and checkout flow. Agents interact through natural language commands that are translated to shopping actions. This environment tests agent capabilities in information seeking, decision-making, and task completion in realistic web scenarios.