Squad V2 Benchmark Aligned Evaluation With Unanswerable Question Handling

via “adversarial unanswerable question generation and validation”

150K reading comprehension questions including unanswerable ones.

Unique: Pioneered adversarial unanswerable questions in QA benchmarks by having crowdworkers explicitly write questions that CANNOT be answered from a passage. This is fundamentally different from randomly sampling unanswerable questions; adversarial construction ensures questions are plausible but genuinely unanswerable.

vs others: More challenging than datasets with random negative examples (e.g., MS MARCO) because adversarial questions require models to understand semantic relevance, not just keyword matching, to distinguish answerable from unanswerable.