ARC (AI2 Reasoning Challenge)

Provides a curated dataset of 7,787 multiple-choice science questions spanning physics, chemistry, biology, and earth science domains at grade-school difficulty levels. The dataset is partitioned into Easy (5,197 questions) and Challenge (2,590 questions) subsets, where Challenge questions are specifically filtered to exclude those solvable by shallow retrieval or word co-occurrence methods, requiring models to perform genuine multi-step scientific reasoning. Enables standardized evaluation of LLM reasoning capabilities against a fixed, reproducible benchmark with known difficulty stratification.

Enables disaggregated evaluation across four science domains (physics, chemistry, biology, earth science) by organizing questions with domain labels, allowing builders to identify which scientific knowledge areas their models struggle with. The dataset structure supports filtering and grouping by domain, producing per-domain accuracy metrics and confusion patterns. This architectural choice surfaces domain-specific reasoning gaps rather than aggregating performance into a single score.

Partitions the dataset into Easy and Challenge subsets with fundamentally different reasoning requirements: Easy questions are solvable through direct retrieval or simple pattern matching, while Challenge questions explicitly exclude such shortcuts and require multi-step inference, knowledge synthesis, and application to novel contexts. This two-tier structure allows builders to measure both baseline knowledge recall and genuine reasoning capability separately, identifying at what reasoning complexity their models begin to fail.

Provides a structured JSON format with consistent question-answer-options schema enabling automated evaluation pipelines. Each question includes the question text, four multiple-choice options (labeled A-D), and a ground-truth answer key. This standardization allows builders to integrate ARC into evaluation frameworks without custom parsing, supporting batch evaluation, metric aggregation, and comparison across model families using a common interface.

Serves as a gold-standard evaluation set for retrieval-augmented generation (RAG) systems by requiring both knowledge retrieval and reasoning steps. Questions cannot be solved by retrieval alone (Challenge set) or by reasoning alone without domain knowledge, making ARC ideal for measuring RAG system effectiveness. Builders can evaluate whether their retrieval component surfaces relevant knowledge and whether their reasoning component correctly applies that knowledge to answer questions.

The ARC dataset includes published baseline results from multiple model families (BERT, RoBERTa, GPT-2, GPT-3, T5, etc.) and reasoning approaches (retrieval-based, word co-occurrence, fine-tuned transformers, few-shot prompting), enabling builders to position their models against known reference points. This allows quantitative comparison without requiring independent implementation of baseline models, accelerating research velocity and enabling fair comparison across different research groups.