MedQA (USMLE)

Provides a standardized benchmark dataset of 12,723 authentic USMLE examination questions spanning Steps 1, 2, and 3, enabling direct assessment of LLM clinical reasoning against the same assessment framework used for medical licensure. The dataset preserves the original multiple-choice format with single correct answers, allowing models to be evaluated on the exact cognitive tasks (diagnosis, treatment planning, pathophysiology, bioethics) that define medical competency. This enables reproducible, calibrated measurement of clinical knowledge acquisition in language models.

Enables evaluation of medical LLMs across three languages (English, Simplified Chinese, Traditional Chinese) using parallel or translated USMLE questions, allowing assessment of whether clinical knowledge transfers across languages or whether language-specific medical terminology and cultural context affect model performance. The dataset structure maintains question-answer alignment across languages, enabling contrastive analysis of multilingual medical reasoning.

Structures questions across USMLE Steps 1, 2, and 3 to assess progressive clinical reasoning complexity: Step 1 tests foundational biomedical knowledge (pathophysiology, pharmacology), Step 2 tests clinical application (diagnosis, management), and Step 3 tests independent clinical judgment (complex cases, ethics, resource allocation). This progression allows evaluation of whether models develop hierarchical clinical reasoning or merely memorize facts, and enables measurement of reasoning capability growth across increasing complexity.

Includes questions explicitly testing bioethics, professional responsibility, and clinical judgment under uncertainty — not just factual medical knowledge. These questions assess whether models understand ethical constraints (informed consent, confidentiality, resource allocation), professional standards, and decision-making in ambiguous scenarios. This capability enables evaluation of whether medical AI systems have acquired not just knowledge but also the ethical reasoning required for clinical practice.

Organizes questions by medical specialty (internal medicine, surgery, pediatrics, obstetrics, psychiatry, etc.), enabling evaluation of whether models have balanced knowledge across clinical domains or exhibit specialty-specific gaps. This allows builders to identify which medical domains a model understands well and which require additional training or caution in deployment. The specialty structure also enables targeted fine-tuning on underperforming domains.

Provides a standardized benchmark aligned with actual medical licensing requirements, enabling healthcare organizations and regulators to assess whether AI systems meet clinical competency thresholds. The dataset includes passing score calibration (GPT-4 achieved passing scores), allowing direct comparison of model performance to human medical professionals. This enables evidence-based regulatory decision-making and clinical deployment authorization.