Humanity's Last Exam vs xCodeEval

Aggregates 2,500 exam questions sourced from 100+ named contributors across academic disciplines through a collaborative curation process. Questions are vetted through a bug bounty program (closed 03/21/2025) that identified and removed searchable/contaminated items, with replacements integrated into the final dataset. The compilation represents a snapshot of expert consensus on difficult, knowledge-testing problems designed to challenge AI reasoning across domains.

Unique: Implements post-hoc contamination mitigation through a formal bug bounty program (03/21/2025) that identified and replaced searchable questions before finalization, addressing a critical gap in benchmark validity that most static benchmarks ignore. The collaborative curation model involves 100+ named contributors from diverse institutions rather than a single lab, creating distributed expertise validation.

vs alternatives: Differs from static benchmarks (MMLU, ARC) by actively removing known contamination via bug bounty rather than assuming training data isolation; differs from rolling benchmarks (HELM) by providing a fixed 2,500-question snapshot with explicit Nature publication (01/28/2026) rather than continuous updates.

Provides HLE-Rolling, a dynamic fork released 10/08/2025 that accepts ongoing question contributions from the community via email submission to agibenchmark@safe.ai. Contributors can propose new exam questions that are integrated into a living version of the benchmark with update logs. This enables continuous evolution of the benchmark as new domains emerge or expert consensus shifts, while maintaining the original 2,500-question snapshot as a fixed reference point.

Unique: Decouples the fixed peer-reviewed benchmark (2,500 questions, Nature publication) from a rolling community version (HLE-Rolling) that accepts contributions via email, enabling continuous evolution without requiring full revalidation. This dual-version approach allows researchers to use the stable snapshot for reproducibility while community members drive innovation in the rolling version.

vs alternatives: Combines the reproducibility of static benchmarks with the adaptability of rolling benchmarks, whereas most benchmarks choose one approach (MMLU is static; HELM is rolling but centrally managed). The email-based contribution system is simpler than GitHub-based workflows but less transparent than formal peer review.

Exposes the 2,500-question benchmark via HuggingFace Datasets library under the dataset ID `cais/hle`, enabling one-line programmatic loading via `load_dataset('cais/hle')`. This integration provides standardized data format compatibility with the HuggingFace ecosystem, allowing researchers to load, filter, and evaluate models using standard HF evaluation frameworks without custom data pipelines. The dataset is versioned and hosted on HuggingFace Hub infrastructure.

Unique: Leverages HuggingFace Datasets' Arrow-backed columnar storage and Hub infrastructure for efficient data loading and versioning, rather than distributing raw JSON/CSV files. This enables automatic caching, version pinning, and compatibility with HF Evaluate and Transformers libraries without custom integration code.

vs alternatives: Faster and more reproducible than downloading raw files from GitHub (no manual versioning); more ecosystem-integrated than providing only a GitHub link, as it works seamlessly with HF Evaluate and other standard tools. However, it locks users into the HF ecosystem and adds a dependency on HF Hub availability.

Provides HLE-Rolling Live Submission Dashboard where researchers can submit model predictions and view real-time rankings. The submission process is email-based (agibenchmark@safe.ai) with an unspecified format and evaluation timeline. The dashboard aggregates results across submitted models and displays comparative performance, enabling researchers to benchmark their models against peers and track progress over time. Submission mechanics, evaluation latency, and result publication policy are not documented.

Unique: Implements a rolling leaderboard tied to HLE-Rolling's dynamic question updates, meaning leaderboard rankings may shift as new questions are added by the community. This differs from static leaderboards (MMLU, ARC) where rankings are stable across evaluation runs, introducing temporal dynamics where older submissions may be re-evaluated against expanded question sets.

vs alternatives: Provides public visibility and competitive incentives for model evaluation, whereas many benchmarks only publish results in papers. However, the email-based submission system is less transparent and scalable than GitHub-based leaderboards (e.g., OpenCompass) or web-based submission portals with automated evaluation.

Implements a formal bug bounty program (closed 03/21/2025) that incentivizes researchers to identify questions in the benchmark that are searchable in public training data or otherwise contaminated. Identified questions are flagged, removed from the final 2,500-question set, and replaced with new questions. This post-hoc contamination mitigation approach addresses a critical validity threat by explicitly removing known leakage risks before publication, rather than assuming training data isolation.

Unique: Formalizes contamination detection as a structured, incentivized process rather than assuming it away or addressing it only in post-hoc analysis. By closing the bug bounty before publication and replacing flagged items, the benchmark provides explicit evidence of contamination awareness and remediation, increasing confidence in validity compared to benchmarks that ignore the issue.

vs alternatives: More rigorous than benchmarks that ignore contamination (MMLU, ARC); less comprehensive than continuous contamination monitoring (HELM's rolling updates). The bug bounty approach is transparent and community-driven but time-limited, whereas continuous monitoring would catch contamination in models trained after the benchmark's publication.

The benchmark is published in Nature (Nature 649, 1139–1146, 01/28/2026), providing formal peer review and editorial validation of the benchmark's methodology, validity, and results. This publication signals that the benchmark has undergone rigorous scrutiny by domain experts and meets standards for reproducibility and scientific rigor. The Nature publication establishes the benchmark as a citable reference point for AI evaluation and provides methodological transparency through the peer-reviewed paper.

Unique: Achieves publication in a top-tier multidisciplinary journal (Nature) rather than a specialized AI conference, signaling that the benchmark's design and validity are of interest to the broader scientific community. This differs from most AI benchmarks (MMLU, ARC, HELM) which are published in AI-specific venues, providing cross-disciplinary validation.

vs alternatives: Nature publication provides higher prestige and broader scientific credibility than conference papers or preprints; however, it also means the benchmark is evaluated against standards for biological, physical, and social sciences, not just AI evaluation practices. The peer review process may be slower and more conservative than rapid iteration in the AI community.

Aggregates exam questions from 100+ named contributors spanning diverse academic institutions and disciplines. The curation process involves distributed expertise validation where questions are proposed by domain experts and vetted through the bug bounty and editorial process. This collaborative approach ensures breadth of coverage across disciplines and reduces single-lab bias compared to benchmarks created by a single research team. Contributor affiliations and discipline distribution are documented but not detailed in available materials.

Unique: Distributes curation across 100+ named contributors from diverse institutions rather than centralizing question creation in a single lab, reducing single-perspective bias and enabling domain-specific expertise validation. The collaborative model is more transparent about contributor identity than benchmarks created by anonymous crowdsourcing or single teams.

vs alternatives: Broader expertise than single-lab benchmarks (MMLU, ARC created by specific teams); more transparent contributor attribution than crowdsourced benchmarks (which often anonymize workers). However, distributed curation may introduce inconsistency in question quality or difficulty compared to centralized editorial control.

Provides a stable, finalized set of 2,500 exam questions (as of 04/03/2025) that serves as the reference benchmark for reproducible evaluation. This fixed snapshot is distinct from the rolling HLE-Rolling version and enables researchers to conduct evaluations that can be exactly reproduced by other teams using the same question set. The snapshot is versioned and published in Nature, establishing it as a canonical reference point for AI evaluation.

Unique: Decouples the fixed reference benchmark (2,500 questions, Nature publication, reproducible) from the rolling version (HLE-Rolling, community contributions, evolving). This dual-version approach allows researchers to use the stable snapshot for reproducible comparisons while the rolling version evolves with community input, balancing reproducibility and adaptability.

vs alternatives: Provides reproducibility guarantees that rolling benchmarks (HELM) cannot offer, since HELM's question set changes over time. However, it sacrifices adaptability compared to rolling benchmarks, potentially becoming outdated as AI capabilities advance. The fixed snapshot is more reproducible than GitHub-based benchmarks without version pinning.

Provides a standardized evaluation framework for code generation models that accepts generated code in 17 programming languages (C, C++, C#, Java, Kotlin, Go, Rust, Python, Ruby, PHP, JavaScript, Perl, Haskell, OCaml, Scala, D, Pascal) and validates correctness through actual execution against unit tests via the ExecEval Docker-based execution engine. Uses a centralized problem definition model with src_uid foreign keys linking generated code to shared problem descriptions and unittest_db.json, enabling consistent evaluation across language variants of the same problem.

Unique: Combines 25M training examples across 7,500 unique problems with an execution-based evaluation pipeline (ExecEval) that actually runs generated code in Docker containers against unit tests, rather than relying on static analysis or string matching. The src_uid linking system creates a normalized data model where problem descriptions and tests are stored once and referenced by all language variants, eliminating duplication and ensuring consistency.

vs alternatives: Larger scale (25M examples vs typical 10-100K) and true execution-based validation across more languages (17 vs 4-6) than HumanEval or CodeXGLUE, with explicit support for code translation and repair tasks beyond generation.

Implements a foreign key linking system where all task-specific datasets (program synthesis, code translation, APR, retrieval) reference shared problem definitions via src_uid identifiers. Problem descriptions and unit tests are stored once in centralized problem_descriptions.jsonl and unittest_db.json files, then linked by src_uid to avoid duplication. The Hugging Face datasets API automatically resolves these links during data loading, returning enriched DatasetDict objects with problem context pre-joined to task examples.

Unique: Uses a normalized relational data model (src_uid as foreign key) for a code benchmark, treating problem definitions as a separate entity layer rather than embedding them in each task dataset. This is more sophisticated than typical flat-file benchmark structures and enables consistent multi-task evaluation on identical problems.

vs alternatives: More efficient than duplicating problem descriptions across 7 task datasets (reduces storage by ~30-40%), and enables automatic link resolution via Hugging Face API unlike manual CSV joins in CodeXGLUE or HumanEval variants.

Provides a Python API for loading xCodeEval datasets from Hugging Face Hub (NTU-NLP-sg/xCodeEval) with automatic src_uid-based linking between task datasets and shared problem definitions. The datasets library handles data downloading, caching, and streaming, while the xCodeEval integration automatically joins task examples with problem_descriptions.jsonl and unittest_db.json using src_uid foreign keys. Returns DatasetDict objects with enriched examples ready for model training or evaluation.

Unique: Integrates xCodeEval with Hugging Face datasets library, providing automatic src_uid resolution and streaming support. Treats data loading as a first-class concern with built-in linking logic, rather than requiring manual JSON parsing.

vs alternatives: More convenient than manual Git LFS downloads because it handles caching and automatic linking, and integrates seamlessly with Hugging Face training pipelines vs custom data loaders.

Provides an alternative data access method using Git LFS for users who prefer direct file access or need selective dataset downloads. Supports cloning the repository with LFS disabled, then pulling specific task files or problem definitions on demand. Useful for custom processing pipelines or environments where Python/Hugging Face is not available, though requires manual src_uid linking to join task examples with problem definitions.

Unique: Provides Git LFS-based alternative to Hugging Face API, enabling direct file access and selective downloads. Requires manual src_uid linking but offers more control over data access patterns.

vs alternatives: More flexible than Hugging Face API for selective downloads and custom pipelines, but requires more manual work for src_uid linking and lacks automatic caching/streaming.

Implements a standardized three-phase evaluation pipeline (Phase 1: Generation, Phase 2: Execution, Phase 3: Metrics) that applies consistently across all 7 tasks (program synthesis, code translation, APR, tag classification, code compilation, NL-code retrieval, code-code retrieval). Phase 1 generates or retrieves code, Phase 2 executes it via ExecEval or computes retrieval metrics, and Phase 3 aggregates results into pass@k, MRR, NDCG, or other task-specific metrics. Enables direct comparison of model performance across tasks.

Unique: Defines a unified three-phase evaluation pipeline that applies to all 7 tasks, treating generation, execution, and metric computation as separate concerns. Enables consistent evaluation methodology across diverse task types (generation, translation, retrieval, classification).

vs alternatives: More comprehensive than task-specific evaluation scripts because it provides a unified framework for all 7 tasks, and enables direct comparison of model performance across different task types.

Evaluates code generation models on the program synthesis task by accepting natural language problem descriptions and generating code solutions in any of 17 languages. The evaluation pipeline (Phase 1: Generation, Phase 2: Execution, Phase 3: Metrics) runs generated code against unit tests via ExecEval, computing pass@k metrics (pass@1, pass@10, etc.) that measure the probability of finding a correct solution within k samples. Supports both single-solution and multi-sample evaluation modes for assessing model reliability.

Unique: Implements a three-phase evaluation pipeline (Generation → Execution → Metrics) with explicit pass@k computation that measures the probability of finding a correct solution within k attempts, rather than just binary pass/fail. Supports multi-sample evaluation across 17 languages with language-specific compiler configurations and timeout handling.

vs alternatives: More rigorous than HumanEval's simple pass@k because it handles language-specific compilation errors and timeouts explicitly, and scales to 25M training examples vs HumanEval's 164 problems.

Evaluates code translation models by accepting source code in one language and generated translations in a target language, then validating functional equivalence through execution against shared unit tests. The translation evaluation pipeline compiles and executes both source and translated code against the same unittest_db.json test cases, comparing outputs to detect translation errors. Supports all 17 language pairs (though not all pairs may have training data) and uses language-specific compiler mappings to handle syntax differences.

Unique: Validates code translation by executing both source and target code against identical unit tests and comparing outputs, ensuring functional equivalence rather than syntactic similarity. Uses language-specific compiler mappings to handle the complexity of 17 different compilation environments and their idiosyncrasies.

vs alternatives: More rigorous than BLEU-score-based translation metrics because it validates actual functional correctness through execution, and covers more language pairs (17 vs typical 2-4) with explicit compiler integration.

Evaluates program repair models by providing buggy code snippets and expecting corrected versions that pass unit tests. The APR evaluation pipeline executes repaired code against unittest_db.json test cases, measuring whether the repair successfully fixes the bug without introducing new failures. Supports repairs across all 17 languages and uses the same execution-based validation as program synthesis, enabling direct comparison of repair quality.

Unique: Treats program repair as an executable task where success is measured by unit test passage, rather than syntactic similarity to reference repairs. Integrates with the same ExecEval pipeline as program synthesis, enabling direct performance comparison between generation and repair models.

vs alternatives: More comprehensive than traditional APR benchmarks (Defects4J, QuixBugs) because it covers 17 languages and 7,500 problems vs 395 Java bugs, and uses consistent execution-based metrics across all repair types.