Adversarially Filtered Commonsense Reasoning Benchmark Construction

via “adversarially-filtered commonsense reasoning benchmark construction”

44K pronoun resolution problems testing commonsense understanding.

Unique: Applies multi-stage adversarial filtering (automated bias detection + human validation) to remove examples solvable via statistical shortcuts, ensuring models must perform genuine semantic reasoning rather than exploiting dataset artifacts like word frequency correlations or syntactic position biases

vs others: More robust than earlier Winograd Schema Challenge (273 examples) by scaling to 44K examples while maintaining adversarial filtering, and more resistant to gaming than unfiltered pronoun resolution datasets like OntoNotes by explicitly removing statistical biases

via “common-sense reasoning on visual scenes”

Real-world visual QA requiring spatial reasoning.

Unique: Evaluates common-sense reasoning on real-world photographs where correct answers require implicit world knowledge rather than explicit visual features, testing whether models have internalized practical understanding during pretraining — architectural choice that assesses reasoning capability beyond visual pattern matching

vs others: More representative of real-world reasoning requirements than visual-only benchmarks, but harder to validate and more prone to annotation bias than benchmarks with objective ground truth

via “grade-school science question benchmark evaluation”

7.8K science questions testing genuine reasoning, not just recall.

Unique: Explicitly designed to filter out questions answerable by retrieval or word co-occurrence — the Challenge subset (2,590 questions) was curated by removing questions that simple baseline methods could solve, ensuring the remaining questions require genuine multi-step reasoning and knowledge application rather than surface-level pattern matching

vs others: More rigorous than generic QA benchmarks because it explicitly excludes questions solvable by shallow methods, making it a stricter test of reasoning; smaller and more focused than MMLU but with deeper curation for reasoning-specific evaluation

via “commonsense reasoning benchmark dataset”

70K commonsense reasoning questions with adversarial distractors.

Unique: Utilizes adversarial filtering to ensure that incorrect options are specifically designed to mislead machines while remaining obvious to humans.

vs others: Offers a unique approach to commonsense reasoning evaluation that combines human-like accuracy with challenging adversarial examples, setting it apart from traditional datasets.

via “multi-step mathematical reasoning benchmark evaluation”

8.5K grade school math problems — multi-step reasoning, verifiable solutions, reasoning benchmark.

Unique: Uses linguistically diverse, human-authored grade school problems (not synthetic) that require genuine multi-step reasoning with basic arithmetic, combined with a standardized answer extraction format (#### delimiter) that enables reproducible evaluation across heterogeneous model outputs

vs others: More challenging than simple arithmetic benchmarks (requires 2-8 reasoning steps) yet more accessible than advanced math benchmarks, making it ideal for measuring practical reasoning improvements in production models

via “commonsense reasoning evaluation”

Commonsense NLI with adversarial context mining

Unique: Utilizes adversarially filtered questions to create plausible distractors, ensuring a more robust evaluation of reasoning capabilities compared to traditional benchmarks.

vs others: More challenging than standard commonsense benchmarks due to its focus on plausible distractors, making it a better test for true understanding.

via “abstract reasoning problem generation”

Abstraction and reasoning corpus for general intelligence

Unique: The design of the problems specifically targets abstract reasoning, distinguishing it from other benchmarks that may not focus on visual inference.

vs others: More focused on abstract reasoning than standard datasets like MNIST, which primarily test recognition rather than inference.

via “commonsense reasoning evaluation through pronoun disambiguation”

Commonsense reasoning with pronoun resolution

Unique: WinoGrande's dataset is uniquely designed to challenge models on their understanding of context and semantics rather than relying on statistical patterns, making it a more rigorous test of reasoning capabilities.

vs others: More comprehensive than traditional benchmarks like Winograd Schema Challenge, as it includes a larger and more diverse set of examples.

via “reasoning capability evaluation”

Subset of BIG-Bench where most models fail

Unique: The curation of tasks specifically targeting reasoning limits rather than general performance allows for a more focused evaluation of model capabilities.

vs others: More targeted than generic benchmarks, as it specifically identifies and tests reasoning weaknesses in models.

via “commonsense-reasoning-benchmark-dataset-loading”

Dataset by Rowan. 3,02,991 downloads.

Unique: Combines video-grounded context from ActivityNet Captions with adversarially-collected wrong answers (via crowdsourcing) to create harder commonsense reasoning tasks than typical multiple-choice datasets; uses HuggingFace's streaming infrastructure for efficient loading of 300K+ examples without requiring full downloads

vs others: Larger and more adversarially-challenging than SWAG (88K examples) with better video grounding than pure text-based commonsense datasets like CommonsenseQA, while maintaining standardized HuggingFace integration for reproducible benchmarking