MT-Bench

MT-Bench evaluates LLM responses across 80 curated multi-turn questions using GPT-4 as an automated judge. The system submits model responses to GPT-4 with structured prompts that assess instruction following, reasoning coherence, and conversation quality, then aggregates scores into comparative rankings. This approach enables large-scale comparative evaluation without human annotation bottlenecks while maintaining consistency through a single judge model.

MT-Bench provides 80 hand-crafted questions distributed across 8 semantic categories (writing, roleplay, reasoning, math, coding, extraction, STEM, knowledge) designed to test diverse LLM capabilities in multi-turn contexts. Questions are structured with explicit turn sequences where later turns depend on earlier context, requiring models to maintain state and refine responses. This stratified approach ensures balanced evaluation across capability domains rather than random sampling.

MT-Bench integrates with Chatbot Arena's Elo rating system to convert pairwise evaluation scores into a unified leaderboard ranking. When GPT-4 judges compare model responses, the scores feed into an Elo algorithm that updates each model's rating based on win/loss/tie outcomes. This approach handles transitive ranking (A > B, B > C implies A > C) and accounts for uncertainty through rating volatility, enabling fair comparison even when models haven't been directly compared.

MT-Bench leverages FastChat's conversation template system to format multi-turn questions consistently across 70+ different LLM architectures (Vicuna, LLaMA, ChatGLM, Falcon, etc.). Each model has a registered template that specifies prompt format, special tokens, and turn delimiters. When evaluating a model on MT-Bench questions, the system automatically applies the correct template, ensuring that differences in evaluation scores reflect model capability rather than prompt formatting artifacts.

MT-Bench evaluation integrates with FastChat's distributed serving infrastructure (controller + model workers) to orchestrate batch evaluation of 80 questions across multiple models in parallel. The controller routes evaluation requests to available model workers, which run inference on their assigned models. Results are collected, deduplicated, and submitted to GPT-4 for judging. This architecture enables scaling evaluation across multiple GPUs/machines without reimplementing model serving.

Chatbot Arena (which uses MT-Bench as one evaluation component) collects human votes on side-by-side model comparisons and aggregates them to calibrate and validate the GPT-4 judge. When users vote on which model response is better, those votes are stored and analyzed to measure agreement between human preferences and GPT-4 scores. This feedback loop enables detection of judge bias and continuous refinement of evaluation rubrics.

MT-Bench's 8-category structure (writing, roleplay, reasoning, math, coding, extraction, STEM, knowledge) enables disaggregated performance analysis where evaluation scores are computed separately per category. This allows identification of which capability domains each model excels or struggles in, rather than a single aggregate score. Researchers can visualize performance profiles showing, e.g., that Model A is strong in coding but weak in reasoning, while Model B shows the opposite pattern.

MT-Bench provides a fixed, versioned set of 80 questions and integrates with FastChat's evaluation tracking to record which judge version (e.g., GPT-4 Turbo vs GPT-4 base) was used for each evaluation. This enables reproducible comparisons: the same questions evaluated with the same judge produce consistent results, and changes in judge version are explicitly tracked. Researchers can compare model A evaluated in month 1 vs month 2 while controlling for judge differences.

MT-Bench provides a unified evaluation interface that works with both local model inference (via FastChat workers) and remote API-based models (OpenAI, Anthropic, etc.). The system abstracts away differences in inference APIs, handling authentication, request formatting, response parsing, and error handling uniformly. This enables fair comparison between open-source models (evaluated locally) and proprietary models (via APIs) in a single benchmark.

MT-Bench ensures reproducibility by using a fixed, versioned set of 80 questions and managing random seeds for model inference (temperature, sampling parameters). The system records evaluation metadata (model version, inference parameters, GPT-4 model version, timestamp) enabling exact reproduction of results. Questions are publicly available, allowing external researchers to verify results or run independent evaluations.