TruthfulQA

Provides a curated dataset of 817 questions specifically engineered to expose when language models reproduce common human misconceptions rather than generate factually correct answers. Questions are distributed across 38 semantic categories (health, law, finance, conspiracy theories, etc.) and paired with reference answers that distinguish between truthful responses and plausible-but-false alternatives that models commonly learn from training data. Evaluation is performed by comparing model outputs against ground-truth labels using both truthfulness scoring (binary/multi-class factual correctness) and informativeness metrics (depth and usefulness of generated content).

Enables disaggregated evaluation of model truthfulness across 38 distinct semantic categories (health, law, finance, politics, conspiracy theories, etc.), allowing developers to identify domain-specific failure modes and knowledge gaps. The dataset structure supports stratified sampling and per-category metric computation, revealing whether a model's truthfulness is uniform across domains or concentrated in certain areas. This architectural design enables fine-grained diagnosis of training data biases and domain-specific hallucination patterns.

Provides reference answers for each question paired with dual evaluation criteria: truthfulness (factual correctness against ground truth) and informativeness (whether the answer provides useful, substantive detail). The dataset includes curated reference answers that serve as ground truth, enabling both automated comparison (via string matching or semantic similarity) and LLM-based judgment. This dual-metric design allows evaluation of the trade-off between accuracy and answer quality, preventing models from gaming the benchmark by providing technically true but useless responses.

Questions are adversarially engineered to target specific common human misconceptions and false beliefs that language models frequently reproduce from training data. Rather than asking generic factual questions, each question is designed to elicit a particular false answer that the model is likely to have learned. This adversarial design pattern enables systematic exposure of model failure modes by directly probing known misconceptions (e.g., 'Do vaccines cause autism?' targets a widespread false belief). The dataset includes questions across health, law, finance, and conspiracy theory domains where misconceptions are most prevalent.

Dataset explicitly covers high-stakes domains (healthcare, law, finance, conspiracy theories) where model hallucination or factual errors could cause real-world harm. The 38 categories are weighted toward safety-critical knowledge areas where false information poses significant risks. This domain selection enables evaluation of model reliability in regulated or high-consequence environments before deployment. The architectural choice to focus on misconception-prone domains rather than general knowledge ensures that evaluation effort is concentrated on areas where model failures are most consequential.

Dataset is hosted on Hugging Face Hub with standardized loading via the `datasets` library, enabling one-line programmatic access and integration into existing ML workflows. The dataset follows Hugging Face conventions (splits, features, metadata) allowing seamless integration with popular evaluation frameworks and model evaluation pipelines. This architectural choice eliminates custom data parsing and enables reproducible, version-controlled evaluation across teams and projects.