AgentBench

Evaluates LLMs as autonomous agents across 8 distinct environments (OS, DB, KG, DCG, LTP, HH, WS, WB) using a standardized Task Interface that defines sample retrieval, execution, and metric calculation. The framework abstracts environment-specific logic behind a common contract, enabling systematic comparison of agent performance across heterogeneous task types with environment-specific startup times (5s-5min) and resource requirements (500MB-15GB). Agents interact with tasks through multi-turn Session management that tracks conversation history and message exchange.

Provides a contract-based Task interface that all benchmark environments implement, defining methods for retrieving sample indices, executing individual samples with agent interactions, and calculating overall performance metrics. The interface abstracts environment-specific logic (game engines, database systems, web simulators) behind common method signatures, enabling the framework to orchestrate agent evaluation without coupling to particular environment implementations. Each task environment implements sample retrieval, step-by-step execution with agent actions, and metric aggregation.

Provides a web shopping task environment where agents interact with a simulated e-commerce platform to complete shopping tasks (product search, comparison, purchase). Agents navigate product catalogs, read descriptions and reviews, manage shopping carts, and complete transactions through a web interface. The environment simulates realistic e-commerce workflows with product filtering, price comparison, and checkout processes. Tasks evaluate agent capabilities in information seeking, decision-making under uncertainty, and multi-step task completion in a complex web environment (~15GB resource requirement).

Provides a web browsing task environment where agents navigate websites to find information and complete web-based tasks. Agents interact with a simulated web browser, following links, reading page content, and performing searches to locate specific information. The environment simulates realistic web navigation with multiple pages, search results, and information density variations. Tasks evaluate agent capabilities in web navigation, information retrieval, and multi-step task completion in open-ended web environments (~1GB resource requirement, ~5min startup).

Provides a household task environment where agents complete domestic tasks in a simulated home environment (based on ALFWorld). Agents interact with a text-based or visual home simulator, manipulating objects, navigating rooms, and completing household chores (cooking, cleaning, organizing). The environment simulates realistic household physics and object interactions, requiring agents to reason about spatial relationships, object properties, and task decomposition. Tasks evaluate agent capabilities in embodied reasoning, multi-step task planning, and interactive problem-solving.

Provides a lateral thinking puzzle task environment where agents solve puzzles requiring creative, non-linear reasoning and constraint satisfaction. Agents interact with a puzzle system that presents scenarios, accepts guesses/hypotheses, and provides feedback on correctness. The environment manages puzzle state, constraint tracking, and solution validation. Tasks evaluate agent capabilities in creative problem-solving, hypothesis generation, constraint reasoning, and iterative refinement. Agents must think beyond obvious solutions and reason about implicit constraints.

Provides a digital card game task environment where agents play strategic card games requiring decision-making, resource management, and opponent modeling. Agents receive game state information (hand, board, opponent state), select actions (play cards, attack, defend), and observe game outcomes. The environment manages game rules, turn order, win conditions, and card interactions. Tasks evaluate agent capabilities in strategic reasoning, resource optimization, and decision-making under uncertainty. Agents must balance multiple objectives and adapt strategies based on game state.

Provides a configuration system that enables users to define task environments, agent parameters, and evaluation assignments through YAML or JSON configuration files. The configuration system abstracts away code-level customization, enabling non-developers to set up benchmarks by editing configuration files. Supports task-specific parameters (environment type, sample count, resource limits), agent-specific parameters (model, temperature, prompt template), and assignment-level parameters (worker count, timeout). Configuration validation ensures correctness before execution.

Defines a standardized Agent interface that abstracts how LLMs and other decision-makers interact with task environments through a Session communication channel. Agents receive observations from tasks, generate actions, and receive feedback in a multi-turn loop. The interface supports both sophisticated LLM-based agents (with prompt engineering, chain-of-thought reasoning) and naive rule-based agents, enabling comparison of different agent architectures. Session management tracks conversation history and message exchange, providing agents with context for decision-making.

Implements a Session abstraction that manages the communication channel between agents and task environments, handling message exchange, conversation history tracking, and state synchronization across multiple turns. Sessions maintain a chronological record of agent observations, actions, and task feedback, enabling agents to make decisions based on accumulated context. The Session interface standardizes how agents receive observations and submit actions, decoupling agent logic from environment-specific communication protocols.

Implements a Task Controller that orchestrates the execution of benchmark tasks across multiple workers, managing resource allocation, task assignment, and result aggregation. The controller uses a Task Assigner to distribute samples across workers and a pool of Task Workers to execute agent-task interactions in parallel. This architecture enables efficient evaluation of agents across large sample sets while managing system resources (memory, CPU, disk) and handling task startup/teardown. The controller coordinates the lifecycle of task environments (initialization, sample execution, metric calculation, cleanup).

Provides a standardized Evaluation Metrics subsystem where each task environment implements domain-specific metric calculation (e.g., success rate for games, SQL correctness for databases, task completion for household tasks). The framework aggregates per-sample metrics into overall performance scores while preserving environment-specific semantics. Metrics are calculated after task execution completes, enabling post-hoc analysis and comparison across agents. The metric interface supports both binary success indicators and continuous performance scores.

Implements a complete Avalon card game environment where LLM agents play a social deduction game requiring strategic reasoning, communication, and deception detection. The environment includes a game engine that manages game state, turn order, voting mechanics, and win conditions, while agents interact through natural language communication and action selection. The Avalon task evaluates agent capabilities in multi-agent strategic reasoning, persuasion, and information inference from incomplete information. Agents must balance exploration (gathering information) with exploitation (making winning moves).

Provides a Linux OS command-line task environment where agents interact with a shell interface to complete system administration and file manipulation tasks. Agents receive shell prompts, issue commands, and observe command output in a multi-turn interaction loop. The environment manages a sandboxed Linux filesystem and command execution, enabling safe evaluation of agent capabilities in command-line reasoning and system administration. Tasks include file operations, text processing, system queries, and scripting.

Provides a database task environment where agents interact with SQL databases to complete data querying and manipulation tasks. Agents receive database schemas, issue SQL queries, and observe query results in a multi-turn loop. The environment manages a sandboxed database instance with predefined schemas and data, enabling evaluation of agent capabilities in SQL reasoning, schema understanding, and query composition. Tasks include data retrieval, aggregation, filtering, and complex joins.

Provides a knowledge graph task environment where agents query and reason over structured knowledge representations to answer questions and complete reasoning tasks. Agents interact with a knowledge graph API, issuing queries to retrieve entities, relationships, and perform multi-hop reasoning. The environment manages a sandboxed knowledge graph with predefined entities and relationships, enabling evaluation of agent capabilities in semantic reasoning, relationship inference, and multi-step knowledge navigation. Tasks include entity lookup, relationship discovery, and transitive reasoning.