Humanity's Last ExamBenchmark40/100 via “fixed 2500-question snapshot for reproducibility”
Hardest exam questions from thousands of experts.
Unique: Decouples the fixed reference benchmark (2,500 questions, Nature publication, reproducible) from the rolling version (HLE-Rolling, community contributions, evolving). This dual-version approach allows researchers to use the stable snapshot for reproducible comparisons while the rolling version evolves with community input, balancing reproducibility and adaptability.
vs others: Provides reproducibility guarantees that rolling benchmarks (HELM) cannot offer, since HELM's question set changes over time. However, it sacrifices adaptability compared to rolling benchmarks, potentially becoming outdated as AI capabilities advance. The fixed snapshot is more reproducible than GitHub-based benchmarks without version pinning.