MATH Benchmark

Loads 12,500 curated competition mathematics problems from AMC 10, AMC 12, AIME, and Math Olympiad sources via the MATHDataset class, which preprocesses problems with optional solution step inclusion and supports multiple tokenization strategies. The dataset is stratified across 7 mathematical subjects (Prealgebra, Algebra, Number Theory, Counting/Probability, Geometry, Intermediate Algebra, Precalculus) enabling subject-specific evaluation and analysis. Problems are stored in structured JSON format with problem statements, solutions, and answer fields, allowing researchers to load subsets by difficulty or subject.

Implements the is_equiv() function in math_equivalence.py that determines whether two mathematical expressions are semantically equivalent by applying a multi-stage string normalization pipeline. The system handles LaTeX formatting, fraction representations, algebraic simplification, numerical precision issues, and common mathematical symbols through regex-based transformations and symbolic comparison. Rather than exact string matching, it normalizes both expressions into canonical forms before comparison, enabling robust answer verification across different notational styles (e.g., '1/2' vs '0.5' vs '\frac{1}{2}').

Provides eval_math_gpt.py with a run_eval() function that evaluates locally-hosted GPT-style language models on MATH problems using configurable beam search and sampling parameters. The evaluation system generates multiple candidate answers per problem (via beam search or temperature-based sampling), compares each against ground truth using the mathematical equivalence system, and aggregates accuracy metrics. Supports both greedy decoding and stochastic sampling strategies, enabling evaluation of model robustness and uncertainty quantification.

Provides evaluate_gpt3.py that interfaces with OpenAI's GPT-3 API for remote model evaluation on MATH problems, handling API authentication, request batching, rate limiting, and result aggregation. The system submits problem statements to the API, collects model-generated solutions, and verifies correctness using the mathematical equivalence system. Implements retry logic and rate-limit handling to manage API quotas and ensure reliable evaluation across large problem sets.

Aggregates per-problem accuracy results across the 7 mathematical subjects (Prealgebra, Algebra, Number Theory, Counting/Probability, Geometry, Intermediate Algebra, Precalculus) to produce subject-specific and overall accuracy metrics. The system computes accuracy rates, confidence intervals, and failure analysis grouped by subject, enabling fine-grained understanding of model strengths and weaknesses across mathematical domains. Supports visualization and reporting of subject-level performance differences.

Analyzes problem difficulty and solution complexity by examining problem metadata (source competition, year, round) and solution characteristics (length, algebraic complexity, required techniques). The system categorizes problems by difficulty tier (e.g., AMC 10 vs AIME vs Olympiad) and enables filtering and analysis of model performance across difficulty levels. Supports identification of which difficulty tiers present the greatest challenges for language models.

Manages dataset splits (train/test/validation) and ensures proper separation to prevent data leakage during model evaluation. The system loads problem subsets based on split configuration, supports multiple split strategies (random, subject-stratified, difficulty-stratified), and validates that evaluation is performed only on designated test sets. Enables reproducible evaluation by supporting fixed random seeds and split versioning.

Extracts and structures solution steps from problem solutions, enabling evaluation of intermediate reasoning quality and step-by-step correctness. The system parses solution text to identify individual steps, validates each step's mathematical correctness, and measures whether models can generate correct intermediate reasoning. Supports evaluation of both final answer accuracy and solution quality (e.g., whether the reasoning path is sound).

Orchestrates batch evaluation of models across all 12,500 problems with built-in result caching and resumption capability. The system manages evaluation state, caches intermediate results to disk, and supports resuming interrupted evaluations without re-running completed problems. Implements progress tracking, logging, and error handling to ensure reliable evaluation across long-running benchmark jobs.

Supports evaluation of multiple models on the same benchmark and generates comparative leaderboards ranking models by accuracy. The system runs evaluations for different models (local, API-based, or hybrid), aggregates results, and produces ranked leaderboards with per-subject accuracy breakdowns. Enables side-by-side comparison of model performance and identification of best-performing models across different mathematical domains.