BIG-Bench Hard (BBH)

Provides curated few-shot chain-of-thought (CoT) exemplars for 23 hard reasoning tasks, enabling models to learn structured step-by-step problem decomposition through in-context learning. Each task includes 3-5 hand-crafted examples showing intermediate reasoning steps, allowing models to adopt explicit reasoning patterns without fine-tuning. The dataset leverages prompt engineering patterns where models observe reasoning trajectories before solving novel instances.

Organizes 23 tasks across distinct reasoning domains (algorithmic, arithmetic, logical, causal, spatial) with consistent evaluation structure, enabling fine-grained analysis of model strengths and weaknesses by reasoning type. Each task is independently evaluable with its own test set and metrics, allowing researchers to identify which reasoning modalities their models excel or fail at. The stratification enables targeted model development and capability analysis.

Designed specifically to evaluate frontier language models (GPT-4, Claude, Llama 2+, etc.) on hard reasoning tasks where initial model performance was below human level, enabling measurement of model improvement over time and comparison of frontier model capabilities. The dataset enables researchers to track whether new model releases improve on hard reasoning and to identify reasoning capabilities that remain unsolved. Results are directly comparable across models because of standardized evaluation infrastructure.

Enables reproducible evaluation across different models and research groups by providing standardized task definitions, test sets, evaluation metrics, and result aggregation. The dataset structure ensures that different teams can run identical evaluations and compare results directly, reducing evaluation variance and enabling fair model comparison. Standardized evaluation infrastructure supports publishing reproducible results and enables meta-analysis across multiple model evaluations.

Includes human rater performance data for all 23 tasks, establishing ground-truth difficulty calibration and enabling measurement of model-vs-human performance gaps. Tasks were specifically selected where initial model performance fell below human median (50th percentile), creating a calibrated hard benchmark. Human baselines enable researchers to quantify progress toward human-level reasoning and identify tasks where models have surpassed human performance.

Provides consistent evaluation infrastructure across 23 heterogeneous reasoning tasks with unified input/output schemas, metrics computation, and result aggregation. Each task includes standardized test sets, answer formats, and evaluation functions, enabling researchers to run comprehensive benchmarks with a single evaluation script. The harness abstracts task-specific complexity and enables reproducible, comparable results across models and research groups.

Includes algorithmic reasoning tasks (e.g., sorting, graph traversal, dynamic programming) that test whether models can learn and apply computational algorithms through few-shot examples. Tasks present problem descriptions and expect models to reason through algorithmic steps, testing whether models can generalize algorithmic patterns beyond memorized examples. This capability isolates algorithmic reasoning from knowledge retrieval or common-sense reasoning.

Includes multi-step arithmetic and mathematical reasoning tasks (e.g., word problems, numerical reasoning, mathematical deduction) that test whether models can perform accurate calculations and apply mathematical reasoning through few-shot examples. Tasks range from basic arithmetic to more complex mathematical inference, isolating numerical reasoning from language understanding. Evaluation measures both intermediate calculation accuracy and final answer correctness.

Includes logical reasoning tasks (e.g., syllogisms, logical deduction, constraint satisfaction) that test whether models can perform formal logical inference through few-shot examples. Tasks present logical premises and expect models to derive correct conclusions, testing whether models can apply logical rules consistently. This capability isolates formal logical reasoning from common-sense reasoning or knowledge retrieval.

Includes causal reasoning tasks that test whether models can identify causal relationships, make causal inferences, and reason about cause-and-effect through few-shot examples. Tasks present scenarios and expect models to identify causal mechanisms or predict causal outcomes, testing whether models can reason about causality beyond correlation. This capability isolates causal reasoning from statistical reasoning or common-sense knowledge.

Includes spatial reasoning tasks (e.g., mental rotation, spatial visualization, geometric reasoning) that test whether models can reason about spatial relationships and visualize spatial configurations through few-shot examples. Tasks present spatial descriptions and expect models to reason about spatial transformations or configurations, testing whether models can build and manipulate mental spatial models. This capability isolates spatial reasoning from visual perception or geometric knowledge.

Enables researchers to experiment with few-shot prompt engineering by providing curated exemplars for each task that can be modified, reordered, or augmented to test prompt sensitivity and optimization strategies. The dataset structure supports prompt template variation, exemplar selection strategies, and in-context learning optimization without requiring task re-annotation. Researchers can measure how prompt engineering choices affect model performance on hard reasoning tasks.