Mathematical Reasoning Evaluation

via “zero-shot mathematical reasoning evaluation”

Zero-shot LLM evaluation for reasoning tasks.

Unique: Implements unified zero-shot evaluation specifically designed to isolate reasoning capability from few-shot learning effects, with multi-format answer extraction that handles LaTeX, symbolic, and natural language mathematical expressions without requiring model-specific output formatting

vs others: Differs from general LLM benchmarks (MMLU, GSM8K) by explicitly removing few-shot examples and standardizing evaluation across mathematical domains, providing cleaner signal for foundational reasoning ability

via “solution step extraction and intermediate reasoning evaluation”

12.5K competition math problems — AMC/AIME/Olympiad level, 7 subjects, standard math benchmark.

Unique: Preserves solution steps as first-class data throughout the evaluation pipeline, enabling evaluation of intermediate reasoning quality rather than just final answers. This supports emerging research on chain-of-thought prompting and interpretable AI reasoning.

vs others: More comprehensive than final-answer-only evaluation because it assesses reasoning quality and interpretability, but requires more manual annotation and is harder to automate than simple answer verification.

via “cross-subdiscipline mathematical reasoning measurement”

Expert-level math problems created by mathematicians.

Unique: Explicitly structures evaluation across four mathematical subdisciplines (number theory, algebra, geometry, analysis) to measure generalization and identify domain-specific reasoning patterns, rather than treating mathematics as a monolithic domain

vs others: Provides subdiscipline-specific performance insights that reveal whether AI reasoning is broadly generalizable or domain-dependent, whereas most benchmarks report aggregate mathematical performance

via “arithmetic and mathematical reasoning evaluation”

23 hardest BIG-Bench tasks where models initially failed.

Unique: Focuses specifically on multi-step arithmetic and mathematical reasoning through few-shot examples, isolating numerical reasoning capability from general language understanding. Tasks test both calculation accuracy and mathematical inference patterns.

vs others: More focused on mathematical reasoning than general reasoning benchmarks; more accessible than formal mathematics verification because it uses natural language problem statements rather than symbolic notation.

via “mathematical reasoning over visual data”

Mistral's 124B multimodal model with vision capabilities.

Unique: Achieves 69.4% on MathVista benchmark (outperforming all tested models) through integrated visual parsing and mathematical reasoning in a single 124B model, without requiring separate symbolic math engines or specialized mathematical libraries

vs others: Outperforms GPT-4o, Gemini-1.5 Pro, and Claude-3.5 Sonnet on MathVista while being available for self-hosted deployment, eliminating API dependency for educational or research mathematical analysis

via “multi-step mathematical reasoning benchmark evaluation”

8.5K grade school math problems — multi-step reasoning, verifiable solutions, reasoning benchmark.

Unique: Uses linguistically diverse, human-authored grade school problems (not synthetic) that require genuine multi-step reasoning with basic arithmetic, combined with a standardized answer extraction format (#### delimiter) that enables reproducible evaluation across heterogeneous model outputs

vs others: More challenging than simple arithmetic benchmarks (requires 2-8 reasoning steps) yet more accessible than advanced math benchmarks, making it ideal for measuring practical reasoning improvements in production models

via “mathematical reasoning with math benchmark performance”

Meta's 70B open model matching 405B-class performance.

Unique: Achieves strong mathematical reasoning performance at 70B parameters through instruction-tuning on mathematical problem-solving datasets, enabling competitive MATH benchmark performance without specialized symbolic reasoning modules

vs others: Provides mathematical reasoning capability comparable to larger closed-source models while remaining open-weight and self-hostable, though without formal verification guarantees of symbolic math systems

via “mathematical reasoning with 96.8% gsm8k accuracy”

Largest open-weight model at 405B parameters.

Unique: 405B parameter scale enables 96.8% GSM8K performance through learned chain-of-thought patterns in transformer architecture, achieving near-human accuracy on grade-school math without external symbolic engines or calculators

vs others: Larger model scale than most open-source alternatives improves mathematical reasoning accuracy; however, lacks symbolic verification that specialized math engines provide, making it suitable for reasoning tasks but not formal proofs

via “mathematical reasoning and step-by-step problem solving”

DeepSeek's 236B MoE model specialized for code.

Unique: Trained on 6 trillion tokens including mathematical reasoning datasets and code-based solutions, enabling both symbolic reasoning and code generation for mathematical problems in a single model without separate math-specific components

vs others: Provides integrated mathematical reasoning and code generation (unlike Copilot which focuses on code) while maintaining open-source weights and supporting local deployment

via “mathematical reasoning and problem-solving”

671B MoE model matching GPT-4o at fraction of training cost.

Unique: Achieves 90.2% on MATH benchmark through MoE architecture that routes mathematical reasoning tokens through specialized expert parameters, enabling efficient scaling of reasoning capability without proportional increase in active parameters per token

vs others: Matches GPT-4o mathematical reasoning performance (90.2% MATH) while using 37B active parameters vs GPT-4o's undisclosed parameter count, reducing inference latency and cost for math-heavy workloads

via “competitive mathematical reasoning with transformer-based arithmetic”

01.AI's bilingual 34B model with 200K context option.

Unique: Achieves competitive mathematical reasoning through general-purpose transformer pretraining without documented chain-of-thought training or specialized math fine-tuning, suggesting strong mathematical pattern learning from raw pretraining data. Supports both English and Chinese mathematical notation and problem-solving.

vs others: Delivers competitive math performance at 34B scale without specialized training overhead, reducing model size and inference cost while maintaining reasonable mathematical reasoning for educational and problem-solving applications.

via “mathematical reasoning with math benchmark 80+ and structured problem-solving”

Alibaba's 72B open model trained on 18T tokens.

Unique: Integrates three distinct reasoning paradigms (CoT for symbolic reasoning, PoT for code-based computation, TIR for external tool orchestration) within single 72B dense model, enabling flexible problem-solving strategies without model switching. 128K context window allows full problem histories and solution verification within single inference call.

vs others: Outperforms Llama 2 70B (significantly lower math performance) and matches Llama 3 70B on general benchmarks while offering specialized math reasoning patterns; Qwen2.5-Math 72B variant provides deeper specialization but general-purpose 72B enables seamless math-to-code-to-text transitions without model switching.

via “mathematical problem solving with symbolic reasoning”

Cost-efficient reasoning model with configurable effort levels.

Unique: Implements specialized mathematical reasoning patterns with step-by-step derivation generation, achieving competition-level math performance through domain-specific training rather than general reasoning

vs others: Matches o3 on mathematical benchmarks at lower cost; outperforms standard LLMs (GPT-4, Claude) on competition-level problems due to reasoning-grade capabilities

via “mathematical reasoning and step-by-step problem solving”

text-generation model by undefined. 1,37,84,608 downloads.

Unique: Qwen2.5-7B-Instruct includes explicit training on mathematical reasoning datasets (including GSM8K, MATH, and proprietary datasets) with emphasis on showing intermediate steps and justifying answers. The instruction-tuning includes prompts that encourage the model to 'think step by step' and 'show your work', which are known to improve mathematical reasoning through in-context learning effects.

vs others: Outperforms base Qwen2.5-7B on mathematical reasoning benchmarks by 15-20% due to instruction-tuning; more accessible than specialized math models (like Minerva) for general-purpose deployment

via “multi-step mathematical reasoning evaluation”

Grade school math problems requiring multi-step reasoning

Unique: GSM8K is specifically curated to include a diverse set of multi-step reasoning problems, making it more targeted than generic math datasets, allowing for precise evaluation of reasoning capabilities in LLMs.

vs others: More focused on multi-step reasoning than other benchmarks like MATH, which may include less structured problems.

via “mathematical reasoning and logic problem evaluation with specialized scoring”

ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个大模型，覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型， 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜，也提供规模超200万的大

Unique: Evaluates mathematical reasoning with 1-5 quality scale for reasoning steps rather than binary correctness, enabling partial credit for correct methodology with computational errors. Combines final answer accuracy with reasoning quality assessment to capture mathematical thinking capability. Includes multi-step reasoning problems and logical inference tasks beyond simple arithmetic.

vs others: More nuanced mathematical assessment than MMLU (binary correctness) and captures reasoning quality vs answer-only evaluation

via “mathematical reasoning and symbolic computation”

Mistral Large — powerful reasoning and instruction-following

via “mathematical-problem-solving-with-symbolic-reasoning”

Gemini 2.5 Pro is Google’s state-of-the-art AI model designed for advanced reasoning, coding, mathematics, and scientific tasks. It employs “thinking” capabilities, enabling it to reason through responses with enhanced accuracy...

Unique: Leverages extended internal reasoning to explore multiple mathematical approaches and verify symbolic manipulations before responding, providing higher confidence in mathematical correctness than models without reasoning capabilities.

vs others: Exceeds GPT-4 and Claude on complex mathematics by using internal reasoning to validate symbolic steps, reducing hallucinated solutions and improving explanation quality for educational use cases.

UGI-Leaderboard — AI demo on HuggingFace

Unique: Isolates mathematical reasoning as a distinct evaluation dimension on the leaderboard, enabling models to be ranked separately on math vs general generation, revealing capability specialization.

vs others: Simpler than running MATH or GSM8K locally with custom evaluation scripts, but less transparent than open-source math benchmarks regarding problem selection and difficulty.

via “mathematical reasoning and symbolic computation”

GLM 4 32B is a cost-effective foundation language model. It can efficiently perform complex tasks and has significantly enhanced capabilities in tool use, online search, and code-related intelligent tasks. It...

Unique: GLM 4 32B includes specialized training on mathematical reasoning datasets, enabling it to show work and explain reasoning — not just generate answers — which is critical for educational and verification use cases

vs others: More cost-effective than Wolfram Alpha for symbolic reasoning while providing better explanations than calculators, though less precise than dedicated symbolic engines for complex expressions