mcp-bench

Evaluates LLM agents across three task complexity tiers (single-server, two-server, three-server) by orchestrating tool discovery, selection, and execution across 28 diverse MCP servers. The framework uses a task execution pipeline that manages persistent MCP server connections via connection pooling, routes tool calls through a schema-aware dispatcher, and measures success via multi-dimensional metrics combining LLM-as-judge scoring with rule-based compliance checks.

Manages long-lived connections to 28 MCP servers using connection pooling (via ServerManagerPersistent) to avoid subprocess spawn overhead per tool call. Executes tool invocations concurrently with server-specific rate limiting and timeout enforcement, routing calls through a schema-aware dispatcher that validates tool parameters against declared MCP schemas before execution.

Provides a curated ecosystem of 28 MCP servers spanning biomedical (BioMCP, Medical Calculator), location services (Google Maps, National Parks), academic research (Call for Papers, Paper Search, Wikipedia), finance (DEX Paprika, OKX Exchange), technology (Hugging Face, NixOS, OpenAPI Explorer), data science (NASA Data, Scientific Computing, Weather), and entertainment (Movie Recommender, Game Trends, Reddit). Each server is pre-configured with tool schemas, rate limits, and authentication, enabling agents to discover and use domain-specific tools.

Implements agent reasoning loops that discover available tools, plan tool sequences to achieve task goals, execute tools, observe results, and adapt plans based on outcomes. Agents maintain conversation history with the LLM, enabling multi-turn reasoning where each tool result informs subsequent planning steps. The executor (agent/executor.py) orchestrates these loops, managing tool invocations, error handling, and termination conditions (max steps, task completion).

Combines LLM-based semantic evaluation (using a judge model to score task completion quality) with rule-based metrics (tool usage patterns, schema compliance, planning effectiveness). The evaluator runs post-execution analysis on agent traces, extracting tool call sequences, measuring planning coherence, and detecting schema violations, then synthesizes scores into a multi-dimensional result set with per-dimension rationale.

Abstracts LLM provider differences (Azure OpenAI, OpenRouter, OpenAI-compatible) behind a unified LLMFactory that returns provider-agnostic Agent instances. Agents use a consistent message-passing interface for tool discovery, planning, and execution, with provider-specific details (API endpoints, authentication, model names) isolated in configuration. Supports streaming and non-streaming modes, automatic retry with exponential backoff, and token counting for cost tracking.

Orchestrates end-to-end benchmark runs via BenchmarkRunner, which loads task definitions from YAML, spawns agent instances per task, collects execution traces and evaluation results, and persists results to structured JSON output. Supports batch execution with configurable parallelism, task filtering by complexity tier, and result aggregation with statistical summaries (mean/median/stddev across tasks).

Discovers available tools by introspecting MCP server manifests (from mcp_servers/commands.json), extracting tool names, parameter schemas, descriptions, and required fields. Validates tool invocations against schemas before execution, detecting missing required parameters, type mismatches, and enum violations. Exposes tool metadata to agents via a unified schema registry, enabling agents to reason about tool capabilities and constraints.

Captures detailed execution traces during agent task execution, recording each tool call (name, parameters, result), agent reasoning steps, latencies, and errors. Traces are structured as JSON with timestamps and server metadata, enabling post-hoc analysis of planning quality, tool selection patterns, and failure modes. Supports both streaming (real-time trace output) and batch (post-execution trace collection) modes.

Organizes benchmark tasks into three complexity tiers based on the number of MCP servers required: single-server (tools from one server), two-server (coordinating tools across two servers), and three-server (managing complex workflows across three servers). Each tier tests different agent capabilities: single-server tests tool selection accuracy, two-server tests cross-domain coordination, three-server tests planning under high complexity. Tasks are tagged with complexity_tier in YAML definitions.

Enables benchmark customization via YAML configuration files (config.yaml, tasks/benchmark_tasks.yaml) with schema validation. Supports configuring LLM providers, MCP servers, task definitions, evaluation parameters, and execution settings without code changes. Configuration is loaded at startup and validated against JSON Schema, enabling early error detection and clear error messages for misconfiguration.

Executes multiple benchmark tasks in parallel using a configurable worker pool (default: 4 workers), respecting per-server rate limits and MCP server concurrency caps. Workers are implemented as async tasks that pull tasks from a queue, execute agents, collect results, and push results to an output queue. Rate limiting is enforced at the server level, preventing any single worker from exceeding server quotas.