Natural Language Model Adversarial Testing

via “adversarial-filtered multiple-choice evaluation”

70K commonsense reasoning questions with adversarial distractors.

Unique: Uses adversarial filtering where distractors are selected based on measured model confusion rather than human-written plausibility, creating a dataset that specifically targets machine weaknesses while maintaining human interpretability. This two-stage LLM-generation + human-validation approach is more scalable than purely human-written distractors while maintaining higher quality than random negatives.

vs others: Harder than SWAG (predecessor) because distractors are adversarially selected for model confusion, and more human-aligned than synthetic reasoning datasets because human accuracy (95.6%) validates that hard-for-models questions remain easy for humans.