BIG-Bench Hard (BBH)

Filters 23 challenging tasks from the original 200+ BIG-Bench tasks using a selection criterion: tasks where language models initially scored below average human rater performance. This curation approach identifies reasoning bottlenecks rather than knowledge gaps, enabling targeted evaluation of model reasoning capabilities. The selection process creates a focused benchmark that isolates genuine reasoning difficulty from task ambiguity or knowledge requirements.

Provides 2-8 few-shot examples per task that demonstrate chain-of-thought (CoT) reasoning patterns — showing intermediate reasoning steps rather than just input-output pairs. These exemplars are structured to guide models toward step-by-step decomposition of reasoning problems. The exemplars are manually curated to illustrate the reasoning strategy most effective for each task type (e.g., breaking arithmetic into sub-steps, listing logical premises before deduction).

Aggregates 23 tasks spanning distinct reasoning domains: algorithmic reasoning (e.g., sorting, graph traversal), multi-step arithmetic, logical deduction, causal judgment, and spatial reasoning. Each domain tests different cognitive capabilities, enabling diagnostic evaluation of which reasoning types models struggle with. The task distribution is designed to avoid clustering in a single reasoning modality, providing a balanced assessment across reasoning categories.

Includes human rater performance scores for each task, enabling direct comparison of model outputs against human reasoning ability. The baseline is computed from multiple human annotators per task, providing a reference point for what constitutes 'solved' reasoning. Models are evaluated on whether they meet, exceed, or fall short of human performance, creating a human-anchored evaluation framework rather than absolute accuracy metrics.

Explicitly excludes tasks that primarily test knowledge retrieval, factual recall, or domain-specific expertise. The filtering process identifies tasks where reasoning ability is the bottleneck, not training data coverage. This is achieved by selecting tasks where model performance correlates with reasoning capability rather than knowledge base size, ensuring the benchmark isolates reasoning from memorization.

Provides all 23 tasks in a consistent JSON format with structured fields: task description, few-shot examples, test instances, expected outputs, and evaluation metrics. This standardization enables programmatic task loading, automated evaluation pipelines, and consistent metric computation across all tasks. The structured format reduces parsing overhead and enables batch evaluation of multiple models against the same task instances.

Distributes the benchmark as a Hugging Face Dataset, enabling seamless integration with the HF ecosystem (transformers, datasets, evaluate libraries). The dataset is versioned, cached locally after first download, and supports streaming for large-scale evaluation. Integration with HF enables one-line loading in Python and automatic compatibility with HF evaluation frameworks, reducing setup friction for researchers.

Provides multiple test instances per task (typically 10-100 examples) rather than single-instance evaluation. This enables statistical significance testing and variance analysis across instances, reducing noise from individual task variations. Batch evaluation allows researchers to compute confidence intervals on model performance and detect whether improvements are statistically significant or within noise margins.

Provides evaluation metrics and task definitions that are model-agnostic — tasks can be evaluated against any model (open-source, proprietary, local, API-based) without model-specific instrumentation. Evaluation is based on comparing model outputs to expected answers using standard metrics (exact match, semantic similarity, reasoning trace validation), not on model internals or architecture. This enables fair comparison across heterogeneous model types and sizes.