Multiple Choice Question Answering Dataset Curation

via “question-answer pair dataset curation and versioning”

Multi-turn conversation benchmark — 80 questions, 8 categories, GPT-4 as judge.

Unique: Explicitly structures questions as multi-turn conversations (not single-turn), with each question containing 2-3 sequential turns that build on prior context. Questions are manually curated by LMSYS researchers rather than automatically generated, ensuring semantic diversity and avoiding trivial or duplicate questions.

vs others: More rigorous than auto-generated benchmarks (HELM uses templates) but smaller in scale; provides explicit multi-turn structure that single-turn benchmarks (MMLU, ARC) cannot evaluate.