Standardized Multiple Choice Evaluation Harness

via “standardized multiple-choice evaluation harness”

7.8K science questions testing genuine reasoning, not just recall.

Unique: Provides a clean, standardized multiple-choice format with unique question identifiers and consistent answer choice ordering, enabling direct integration with evaluation frameworks like lm-eval, vLLM's evaluation suite, and Hugging Face's evaluation harness without custom parsing or normalization

vs others: More standardized than ad-hoc science QA datasets because it enforces consistent formatting; more reproducible than datasets with variable question structures or answer choice counts