lm-evaluation-harness

Provides a registry-based abstraction layer that instantiates language models from 25+ backends (HuggingFace, vLLM, OpenAI, Anthropic, local Ollama, etc.) through a single Python API. The registry pattern decouples task definitions from model implementations, allowing users to swap backends without changing evaluation code. Each backend implements a common interface supporting loglikelihood scoring and text generation with automatic tokenization, BOS token handling, and context window management.

Enables users to define evaluation tasks declaratively via YAML configuration files with support for Jinja2 templating, task inheritance, and document processing. Tasks specify prompts, few-shot examples, metrics, and answer extraction logic without writing Python code. The TaskManager loads YAML configs, resolves inheritance chains, and instantiates Task objects that generate evaluation requests. This approach separates task logic from evaluation infrastructure, allowing non-engineers to create benchmarks.

Enables grouping of related tasks into benchmark suites (e.g., MMLU, BigBench, HELM) with aggregated metrics and reporting. Suites can be defined in YAML or Python, with support for task groups, weighted aggregation, and suite-level metrics. The system computes both per-task and suite-level results, with confidence intervals propagated through aggregation. Supports standard NLP benchmarks, multilingual benchmarks, robustness frameworks (SCORE), and custom suites.

Allows advanced users to define evaluation tasks as Python classes extending the Task base class, with custom metric functions and request generation logic. Custom tasks can implement arbitrary evaluation logic beyond YAML capabilities, including complex metrics, multi-stage evaluation, and dynamic request generation. Metrics are registered in a global registry and can be reused across tasks. This provides maximum flexibility for researchers designing novel evaluation approaches.

Abstracts tokenizer differences across models by providing a unified tokenization interface that handles special tokens, padding, and attention masks consistently. The system automatically selects the correct tokenizer for each model backend and applies model-specific token handling (e.g., BOS token prepending for certain models). This enables fair comparison across models with different tokenization schemes, which would otherwise produce different loglikelihood scores for identical prompts.

Provides a comprehensive CLI (lm_eval/__main__.py) that accepts task names, model names, and evaluation parameters as command-line arguments. Supports task filtering (e.g., 'mmlu_*' to run all MMLU variants), model specification with backend selection, and output format configuration. The CLI integrates all framework capabilities (batching, caching, distributed evaluation, logging) without requiring Python code, making the framework accessible to non-programmers.

Enables creation of custom benchmark suites by composing multiple tasks and aggregating their metrics into a single leaderboard score. The system supports weighted aggregation (e.g., MMLU counts more than HellaSwag), per-task metric selection, and hierarchical grouping (e.g., 'reasoning' group contains multiple reasoning tasks). Leaderboard scores are computed with optional normalization and ranking.

Implements configurable few-shot sampling strategies that select examples from the training set to include in prompts. Supports random sampling, stratified sampling (balanced across classes), and deterministic seeding for reproducibility. The system caches sampled examples to avoid recomputation and integrates with the request generation pipeline to prepend examples to each evaluation instance. Sampling respects task-specific constraints (e.g., max tokens, example diversity).

Generates two types of evaluation requests from task definitions: (1) loglikelihood scoring, which computes the model's probability of correct answers given prompts, and (2) text generation, which samples model outputs and compares them to references. The request generator creates batches of requests optimized for the target model backend, handling tokenization, padding, and attention mask generation. Requests are cached to avoid recomputation across multiple evaluation runs.

Computes evaluation metrics (accuracy, F1, BLEU, ROUGE, etc.) from model predictions and references, with automatic bootstrapped confidence interval calculation. The metrics system supports both built-in metrics (via lm_eval/api/metrics.py) and custom metric functions. Bootstrapping resamples predictions with replacement to estimate metric variance and generate 95% confidence intervals, providing statistical rigor beyond point estimates. Metrics are aggregated at task and suite levels.

Enables evaluation across multiple GPUs and distributed systems using PyTorch's DistributedDataParallel or manual batching strategies. The evaluator automatically partitions tasks across devices, balances load based on task complexity, and aggregates results. Supports both data parallelism (same model on multiple GPUs) and model parallelism (model sharded across GPUs). Caching prevents redundant computation across devices, and results are synchronized before final aggregation.

Handles formatting of multi-turn conversations using model-specific chat templates (e.g., ChatML, Llama 2 chat format). The system applies templates to convert task prompts into properly formatted chat messages, handling role assignment (user/assistant), special tokens, and message ordering. Templates are loaded from HuggingFace model configs or defined in task YAML. This enables evaluation of instruction-tuned and chat models with their native prompt formats.

Extracts model answers from generated text using task-specific filters and parsers. Supports regex-based extraction, JSON parsing, multiple-choice option selection, and custom Python functions. Filters handle common cases like extracting the first line, finding the last number, or parsing structured outputs. This decouples answer extraction logic from metric computation, allowing flexible handling of different model output formats.

Logs evaluation results to multiple destinations simultaneously: JSON files, Weights & Biases, HuggingFace Hub, Zeno visualization platform, and custom sinks. The EvaluationTracker (lm_eval/loggers/evaluation_tracker.py) manages result aggregation and formatting for each sink. Results include metrics, confidence intervals, task metadata, model info, and evaluation parameters. Logging is asynchronous to avoid blocking evaluation, and results are cached to enable resumable evaluations.

Implements a multi-level caching system that stores model outputs, loglikelihoods, and metrics to avoid redundant computation. Caches are keyed by model name, task name, and request hash, enabling result reuse across evaluation runs. The system supports both in-memory and disk-based caches, with automatic cache invalidation when task definitions change. Caching is transparent to users and significantly reduces evaluation time for repeated benchmarks.