HELM

Evaluates LLMs against a curated suite of 42 diverse scenarios (e.g., question answering, summarization, toxicity detection, machine translation) using a unified evaluation harness that normalizes inputs, runs inference, and collects outputs in a standardized format. Each scenario is implemented as a pluggable adapter that handles scenario-specific preprocessing, prompt templating, and metric computation, enabling consistent cross-model comparison across heterogeneous task types.

Computes seven distinct metric families for each scenario, each targeting a different dimension of model quality. Accuracy measures correctness; calibration measures confidence alignment; robustness measures performance under input perturbations (typos, paraphrases); fairness measures performance parity across demographic groups; bias measures stereotypical associations; toxicity measures harmful output generation; efficiency measures latency and token cost. Each metric is computed using scenario-specific logic (e.g., F1 for classification, BLEU for generation, toxicity classifier for safety) and aggregated into a unified scorecard.

Provides web-based interactive dashboards for exploring evaluation results, including scenario-level performance tables, metric comparison charts, demographic breakdowns, and robustness analysis. Users can filter by model, scenario, metric, or demographic group; drill down from aggregate metrics to individual predictions; and export results in multiple formats (CSV, JSON, HTML). Dashboards are generated automatically from evaluation results and hosted on the HELM website for public access.

Ensures reproducibility by versioning scenario definitions, prompt templates, and evaluation code; archiving evaluation results with metadata (model version, evaluation date, hardware configuration); and enabling result replication by re-running evaluations with the same code and data. Evaluation runs are tagged with unique identifiers and stored in a results database, enabling tracking of model performance over time and comparison of results across different evaluation runs.

Manages a library of prompt templates for each scenario, supporting multiple prompt variations (e.g., few-shot vs zero-shot, different instruction phrasings, different example selections) to measure prompt sensitivity. Templates are parameterized (e.g., {instruction}, {examples}, {input}) and instantiated per test instance. The framework tracks which template variant was used for each evaluation run, enabling analysis of prompt robustness and comparison of prompt engineering strategies across models.

Aggregates evaluation results across multiple models and scenarios to produce comparative rankings and performance tables. Computes aggregate metrics (e.g., average accuracy across scenarios, weighted by scenario importance) and statistical significance tests (e.g., paired t-tests, bootstrap confidence intervals) to determine whether performance differences are statistically meaningful or due to random variation. Produces interactive dashboards and downloadable result tables enabling side-by-side model comparison.

Analyzes model performance across demographic groups (e.g., gender, race, age, nationality) by computing per-group metrics and detecting performance disparities. For scenarios with demographic annotations, computes group-level accuracy, calibration, and other metrics, then compares across groups to identify fairness issues (e.g., 'model achieves 85% accuracy for male subjects but 72% for female subjects'). Produces fairness reports highlighting disparities and potential sources of bias.

Evaluates model robustness by running inference on perturbed versions of test inputs (e.g., typos, paraphrases, negations, entity substitutions) and comparing performance to clean inputs. Perturbations are generated using rule-based transformations (e.g., random character swaps, synonym replacement) or learned models (e.g., paraphrase generators). Robustness is measured as the performance drop under perturbation, enabling identification of models that degrade gracefully vs catastrophically under distribution shift.

Evaluates model safety by measuring the frequency and severity of toxic, harmful, or unsafe outputs generated by the model. Uses external toxicity classifiers (e.g., Perspective API, local toxicity models) to score model outputs for toxicity, bias, identity attacks, insults, profanity, and other harmful content. Aggregates toxicity scores across scenarios to produce safety metrics (e.g., 'percentage of outputs flagged as toxic', 'average toxicity score'). Enables comparison of safety across models and identification of scenarios that trigger unsafe outputs.

Measures model efficiency by collecting latency (time to generate output), throughput (tokens per second), and token cost (API pricing) during evaluation. Latency is measured end-to-end (prompt + generation time) and broken down by component (prompt processing vs generation). Token cost is computed from model pricing and token counts. Efficiency metrics are aggregated per scenario and per model, enabling cost-performance tradeoff analysis (e.g., 'model A is 2% more accurate but 3x more expensive').

Measures model calibration by comparing predicted confidence scores to actual accuracy. For models that output confidence scores (e.g., probability distributions), computes calibration metrics (e.g., expected calibration error, Brier score) that quantify the gap between confidence and correctness. Well-calibrated models have high confidence when correct and low confidence when incorrect; poorly calibrated models may be overconfident or underconfident. Enables identification of models suitable for high-stakes applications where confidence estimates are critical.

Provides a modular scenario library with 42 pre-built scenarios covering diverse tasks (QA, summarization, translation, toxicity detection, etc.). Each scenario is implemented as a pluggable module defining input/output format, evaluation metrics, and optional prompt templates. Enables users to add custom scenarios by implementing a standard scenario interface, allowing evaluation of domain-specific tasks. Scenarios are versioned and documented to ensure reproducibility and clarity.