few-shot multitask evaluation across 57 knowledge domains, prompt generation with few-shot example formatting, context-aware prompt truncation via bpe tokenization, model calibration measurement across confidence metrics, hierarchical subject organization and result aggregation, standardized evaluation harness with reproducible model testing, structured subject category taxonomy and hierarchical organization

MMLU

BenchmarkFree

57-subject knowledge benchmark — 15K+ questions across STEM, humanities, professional domains.

Open Source

/ 100

7 capabilities

Capabilities7 decomposed

few-shot multitask evaluation across 57 knowledge domains

Medium confidence

Executes standardized few-shot prompting evaluation on language models across 57 subjects (STEM, humanities, social sciences, professional) by constructing few-shot prompts with 5 example question-answer pairs per subject, then measuring accuracy on held-out test sets. The system uses a hierarchical subject organization (e.g., STEM → physics → high school physics) and aggregates results at subject, category, and overall levels to produce granular performance metrics.

Solves for

I need to benchmark my LLM against a standard that covers breadth of knowledge across multiple domainsI want to understand where my model performs well and poorly across different knowledge categoriesI need reproducible evaluation results that are comparable to published leaderboards and research papers

Best for

LLM researchers evaluating foundation models and fine-tuned variants

ML engineers comparing model performance before/after training or instruction-tuning

Teams building general-purpose AI systems that need broad knowledge coverage validation

Requires

Language model with API or local inference capability

MMLU dataset (15,908 questions across 57 subjects) — available in hendrycks/test repository

Python 3.6+ for running evaluation scripts

Limitations

Multiple-choice format doesn't capture reasoning depth or explain-ability — models can guess correctly without understanding

57 subjects provide breadth but limited depth per subject (typically 50-100 questions per subject)

Few-shot evaluation (5 examples) may not reflect zero-shot or many-shot performance patterns

What makes it unique

Organizes 15,908 questions hierarchically across 57 subjects with standardized few-shot prompting (5 examples per subject) and aggregates results at multiple granularity levels (subject, category, overall), enabling both broad coverage assessment and fine-grained domain analysis in a single evaluation run

vs alternatives

Broader coverage than domain-specific benchmarks (57 subjects vs 1-5) and more standardized than ad-hoc evaluation, making it the de facto general knowledge benchmark for LLM comparison in research and industry

prompt generation with few-shot example formatting

Medium confidence

Constructs few-shot prompts by formatting subject name, selecting 5 in-context examples from the training set, and appending the test question with multiple-choice options. The system implements format_subject() to normalize subject names, format_example() to structure each example as 'Question: ... Options: A) ... B) ... C) ... D) ... Answer: X', and gen_prompt() to concatenate examples with the target question. This approach ensures consistent prompt structure across all 57 subjects and enables reproducible few-shot evaluation.

Solves for

I need to generate consistent few-shot prompts for evaluating models on a specific subjectI want to ensure prompt formatting doesn't introduce bias or inconsistency across different subjectsI need to understand what examples are being used to evaluate my model's few-shot learning capability

Best for

Researchers studying few-shot learning behavior across knowledge domains

Teams implementing MMLU evaluation in custom evaluation pipelines

Developers extending MMLU with custom subjects or prompt templates

Requires

MMLU dataset with train/test split per subject

Subject name mapping (e.g., 'abstract_algebra' → 'Abstract Algebra')

Python 3.6+ with string formatting capabilities

Limitations

Fixed 5-example few-shot format — no support for zero-shot or variable-shot evaluation without code modification

Example selection is deterministic (first 5 training examples) — no randomization or stratified sampling

Prompt format is rigid (Question → Options → Answer) — no support for alternative prompt templates or chain-of-thought formatting

What makes it unique

Implements standardized prompt formatting functions (format_subject, format_example, gen_prompt) that ensure consistent structure across all 57 subjects, enabling reproducible few-shot evaluation and reducing prompt-induced variance in model performance measurement

vs alternatives

More reproducible than manual prompt engineering and more standardized than ad-hoc formatting, ensuring that performance differences reflect model capability rather than prompt variation

context-aware prompt truncation via bpe tokenization

Medium confidence

Truncates prompts to fit within model context windows using Byte Pair Encoding (BPE) tokenization. The crop.py system encodes prompts to BPE tokens, truncates to a maximum of 2048 tokens, and decodes back to text while preserving semantic coherence. This approach automatically downloads encoder resources (e.g., GPT-2 tokenizer) if not available locally and ensures prompts fit within typical model context limits without manual length estimation.

Solves for

I need to ensure prompts fit within my model's context window without manual length calculationI want to automatically handle long prompts by truncating them intelligently rather than failingI need consistent tokenization across different models that may use different tokenizers

Best for

Evaluating models with limited context windows (e.g., older models, edge deployments)

Automated evaluation pipelines that need to handle variable-length prompts robustly

Researchers studying the impact of context length on few-shot learning performance

Requires

Python 3.6+ with tiktoken or equivalent BPE encoder library

Internet access to download encoder resources on first run (cached locally thereafter)

Text input (prompt string)

Limitations

Fixed 2048-token limit — no support for dynamic limits based on model capabilities

BPE tokenization may not align with model's actual tokenizer (e.g., models using SentencePiece or Tiktoken)

Truncation is lossy — removes examples or question context, potentially degrading evaluation validity

What makes it unique

Implements automatic BPE-based prompt truncation with local caching of encoder resources, enabling context-aware evaluation without manual prompt length management or model-specific tokenizer configuration

vs alternatives

More robust than character-count-based truncation (which doesn't account for tokenization) and more general than model-specific truncation (which requires per-model configuration)

model calibration measurement across confidence metrics

Medium confidence

Measures how well-calibrated model predictions are using multiple calibration metrics: Expected Calibration Error (ECE), Static Calibration Error (SCE), Root Mean Square Calibration Error (RMSCE), Adaptive Calibration Error (ACE), and Threshold Adaptive Calibration Error (TACE). The calib_tools.py system supports different binning schemes (uniform, adaptive) and normalization methods, enabling analysis of whether model confidence scores align with actual accuracy across prediction classes. This is critical for understanding model reliability beyond raw accuracy.

Solves for

I need to understand if my model's confidence scores are reliable indicators of correctnessI want to measure whether my model is overconfident or underconfident across different prediction classesI need to compare calibration quality across different models or training approaches

Best for

ML engineers building production systems where confidence scores drive downstream decisions (e.g., routing to human review)

Researchers studying model reliability and uncertainty quantification

Teams evaluating whether fine-tuning or instruction-tuning improves model calibration

Requires

Model predictions with confidence scores (probabilities for each class)

Ground truth labels for evaluation set

Python 3.6+ with numpy for metric computation

Limitations

Requires model to output confidence scores or probabilities — doesn't work with models that only output discrete answers

Calibration metrics are aggregate statistics — don't identify which specific classes or domains are miscalibrated

Different metrics (ECE, SCE, ACE) can rank models differently — no consensus on which metric is most important

What makes it unique

Implements five distinct calibration metrics (ECE, SCE, RMSCE, ACE, TACE) with configurable binning schemes and normalization methods, enabling comprehensive analysis of model confidence calibration beyond simple accuracy measurement

vs alternatives

More comprehensive than single-metric calibration (e.g., ECE alone) and more flexible than fixed binning schemes, allowing researchers to identify calibration issues across different granularities and binning strategies

hierarchical subject organization and result aggregation

Medium confidence

Organizes 57 subjects into a hierarchical taxonomy (e.g., STEM → Physics → High School Physics) and aggregates evaluation results at multiple levels: per-subject accuracy, per-category accuracy (e.g., all STEM subjects), and overall benchmark accuracy. The system uses categories.py to define the hierarchy and evaluate_flan.py to compute aggregated metrics, enabling both fine-grained analysis (which specific subjects are weak) and high-level comparison (overall model capability). This hierarchical structure mirrors how knowledge is organized in educational systems.

Solves for

I need to understand my model's performance across different knowledge domains, not just an overall scoreI want to identify which subject areas my model struggles with to guide further training or fine-tuningI need to compare models at different levels of granularity (overall vs category vs subject)

Best for

Researchers analyzing model strengths and weaknesses across knowledge domains

Teams building domain-specific models and wanting to validate breadth of knowledge

Educators or curriculum designers studying how LLMs understand different subjects

Requires

MMLU dataset with subject labels and category mappings

Python 3.6+ with dict/list operations for aggregation

Per-subject accuracy scores from evaluation

Limitations

Subject categorization is fixed — no support for custom hierarchies or alternative taxonomies

Aggregation is simple averaging — doesn't weight subjects by importance or difficulty

57 subjects provide broad coverage but uneven depth (some subjects have 50 questions, others 100+)

What makes it unique

Implements hierarchical subject organization (57 subjects grouped into 4 major categories: STEM, humanities, social sciences, other) with multi-level result aggregation, enabling both granular subject-level analysis and high-level category comparison in a single evaluation framework

vs alternatives

More structured than flat subject lists and more informative than single overall scores, enabling researchers to identify domain-specific weaknesses and guide targeted model improvements

standardized evaluation harness with reproducible model testing

Medium confidence

Provides a complete evaluation harness (evaluate_flan.py) that orchestrates the entire MMLU evaluation workflow: loading dataset, generating few-shot prompts, querying models, collecting predictions, computing accuracy, and aggregating results. The main() function coordinates these steps with configurable parameters (model selection, number of examples, output paths), ensuring reproducible evaluation across different models and runs. This harness abstracts away implementation details and provides a standard interface for model evaluation.

Solves for

I want to evaluate my model on MMLU with minimal custom codeI need reproducible evaluation results that match published benchmarksI want to compare multiple models using the same evaluation protocol

Best for

ML engineers and researchers evaluating models without deep benchmark expertise

Teams integrating MMLU evaluation into CI/CD pipelines or automated testing

Researchers publishing results that need to be reproducible and comparable to prior work

Requires

MMLU dataset (hendrycks/test repository)

Model implementation (FLAN or compatible interface)

Python 3.6+ with file I/O and CSV writing capabilities

Limitations

Harness is specific to FLAN models — extending to other model families requires code modification

No built-in support for batch evaluation or distributed evaluation across multiple GPUs/TPUs

Results are written to CSV files — no structured output format (JSON, database) for programmatic access

What makes it unique

Provides a complete, self-contained evaluation harness that handles dataset loading, prompt generation, model querying, result collection, and aggregation in a single orchestrated workflow, eliminating the need for custom evaluation code

vs alternatives

More complete than individual evaluation functions and more reproducible than manual evaluation scripts, enabling consistent benchmarking across teams and time periods

structured subject category taxonomy and hierarchical organization

Medium confidence

Defines and maintains a hierarchical taxonomy of 57 subjects organized into 4 high-level categories (STEM, humanities, social sciences, professional). The categories.py module encodes this taxonomy as a structured data structure (likely a dictionary or class hierarchy) that maps subjects to categories, enabling consistent categorization across the evaluation pipeline. This taxonomy is used throughout the evaluation process for subject-level result aggregation, category-level analysis, and leaderboard organization.

Solves for

Organize 57 subjects into meaningful high-level categories for performance analysis and reportingEnable category-level performance comparison (e.g., STEM vs humanities) to identify knowledge distribution patternsMaintain consistent subject-to-category mapping across all evaluation runs and publicationsSupport custom analysis and filtering by category or subject

Best for

Researchers analyzing model performance patterns across knowledge domains

Teams publishing MMLU results with category-level breakdowns

Developers building analysis tools that need subject-to-category mappings

Requires

categories.py file with subject-to-category mappings

Python 3.7+ for importing and using taxonomy

Limitations

Taxonomy is fixed and immutable — cannot add new subjects or reorganize categories without modifying source code

Category definitions are coarse-grained (4 categories) — may obscure fine-grained performance patterns within categories

No weighting or importance ranking — all subjects treated equally in aggregation

What makes it unique

Encodes a structured taxonomy of 57 subjects into 4 categories as a centralized, reusable data structure (categories.py), enabling consistent categorization across all evaluation and analysis code. This separation of taxonomy definition from evaluation logic allows researchers to analyze results at multiple levels of granularity without duplicating category mappings.

vs alternatives

Provides a centralized, version-controlled taxonomy compared to ad-hoc category definitions scattered across analysis scripts, ensuring consistency and enabling reproducible category-level analysis across publications.

Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.

Related Artifactssharing capabilities

Artifacts that share capabilities with MMLU, ranked by overlap. Discovered automatically through the match graph.

Model53

Qwen3-8B

text-generation model by undefined. 1,00,18,533 downloads.

few-shot in-context learning for task adaptation

1 shared capability

Model24

MiniMax: MiniMax M2.1

MiniMax-M2.1 is a lightweight, state-of-the-art large language model optimized for coding, agentic workflows, and modern application development. With only 10 billion activated parameters, it delivers a major jump in real-world...

prompt-optimization-and-few-shot-learning

1 shared capability

Model23

OpenAI: GPT-5.2

GPT-5.2 is the latest frontier-grade model in the GPT-5 series, offering stronger agentic and long context perfomance compared to GPT-5.1. It uses adaptive reasoning to allocate computation dynamically, responding quickly...

few-shot-learning-with-in-context-examples

1 shared capability

Model50

Qwen2.5-0.5B-Instruct

text-generation model by undefined. 61,45,130 downloads.

few-shot prompt adaptation via in-context learning

1 shared capability

Model21

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (GPT-NeoX)

* ⭐ 04/2022: [PaLM: Scaling Language Modeling with Pathways (PaLM)](https://arxiv.org/abs/2204.02311)

few-shot and zero-shot task adaptation

1 shared capability

Model52

Qwen2.5-3B-Instruct

text-generation model by undefined. 92,07,977 downloads.

few-shot learning via in-context examples

1 shared capability

Best For

✓LLM researchers evaluating foundation models and fine-tuned variants
✓ML engineers comparing model performance before/after training or instruction-tuning
✓Teams building general-purpose AI systems that need broad knowledge coverage validation
✓Researchers studying few-shot learning behavior across knowledge domains
✓Teams implementing MMLU evaluation in custom evaluation pipelines
✓Developers extending MMLU with custom subjects or prompt templates
✓Evaluating models with limited context windows (e.g., older models, edge deployments)
✓Automated evaluation pipelines that need to handle variable-length prompts robustly

Known Limitations

⚠Multiple-choice format doesn't capture reasoning depth or explain-ability — models can guess correctly without understanding
⚠57 subjects provide breadth but limited depth per subject (typically 50-100 questions per subject)
⚠Few-shot evaluation (5 examples) may not reflect zero-shot or many-shot performance patterns
⚠No evaluation of reasoning steps or intermediate work — only final answer correctness
⚠Fixed 5-example few-shot format — no support for zero-shot or variable-shot evaluation without code modification
⚠Example selection is deterministic (first 5 training examples) — no randomization or stratified sampling

Requirements

Language model with API or local inference capabilityMMLU dataset (15,908 questions across 57 subjects) — available in hendrycks/test repositoryPython 3.6+ for running evaluation scriptsModel must support prompt-based inference (text input → text output)MMLU dataset with train/test split per subjectSubject name mapping (e.g., 'abstract_algebra' → 'Abstract Algebra')Python 3.6+ with string formatting capabilitiesPython 3.6+ with tiktoken or equivalent BPE encoder library

Input / Output

Accepts: text prompts (few-shot examples + test question), multiple-choice questions with 4 options (A, B, C, D), subject name (string, e.g., 'physics'), training examples (list of dicts with 'question', 'choices', 'answer' keys), test question (dict with same structure), prompt text (string of arbitrary length), predictions (list of predicted class labels), confidence scores (list of floats 0-1 for each prediction), ground truth labels (list of true class labels), per-subject accuracy scores (dict mapping subject name to accuracy %), subject-to-category mapping (dict defining hierarchy), model identifier or path, dataset path, configuration parameters (num_examples, output_path, etc.), Subject name (string, e.g., 'abstract_algebra')

Produces: single-character answer (A/B/C/D), accuracy scores per subject (0-100%), aggregated accuracy across categories and overall benchmark, formatted prompt string (text), prompt with embedded examples and target question, truncated prompt text (string, max 2048 BPE tokens), calibration metrics (dict with ECE, SCE, RMSCE, ACE, TACE values), calibration plots (optional visualization of confidence vs accuracy), per-subject accuracy (dict), per-category accuracy (dict), overall benchmark accuracy (float 0-100), CSV files with per-subject and aggregated results, accuracy metrics (per-subject, per-category, overall), Category name (string, e.g., 'STEM'), List of subjects in a category, Complete taxonomy as structured data

UnfragileRank

Adoption70%(25% weight)

Quality85%(35% weight)

Ecosystem30%(15% weight)

Match Graph25%(20% weight)

Freshness100%(5% weight)

UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.

Type: Benchmark

7 capabilities

Visit MMLU→

About

Massive Multitask Language Understanding. 15,908 questions across 57 subjects (STEM, humanities, social sciences, professional). Tests broad knowledge and problem-solving. The most widely reported general LLM benchmark.

Alternatives to MMLU

v087Product

AI UI generator by Vercel — creates production-quality React/Next.js components from natural language descriptions.

Compare →

Framer82Product

AI-powered website design and publishing — generates responsive, professionally designed sites from descriptions.

Compare →

Midjourney79Product

AI image generation — artistic high-quality outputs, Discord bot, photorealistic V6 model.

Compare →

xCodeEval67Benchmark

Multilingual code evaluation across 17 languages.

Compare →

Are you the builder of MMLU?

Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.

Claim this artifact →Verification via email

Get the weekly brief

New tools, rising stars, and what's actually worth your time. No spam.

Data Sources

seed developer essentials

Looking for something else?

Search →

Capabilities7 decomposed

few-shot multitask evaluation across 57 knowledge domains

Medium confidence

Solves for

Best for

LLM researchers evaluating foundation models and fine-tuned variants

ML engineers comparing model performance before/after training or instruction-tuning

Teams building general-purpose AI systems that need broad knowledge coverage validation

Requires

Language model with API or local inference capability

MMLU dataset (15,908 questions across 57 subjects) — available in hendrycks/test repository

Python 3.6+ for running evaluation scripts

Limitations

Multiple-choice format doesn't capture reasoning depth or explain-ability — models can guess correctly without understanding

57 subjects provide breadth but limited depth per subject (typically 50-100 questions per subject)

Few-shot evaluation (5 examples) may not reflect zero-shot or many-shot performance patterns

What makes it unique

vs alternatives

prompt generation with few-shot example formatting

Medium confidence

Solves for

Best for

Researchers studying few-shot learning behavior across knowledge domains

Teams implementing MMLU evaluation in custom evaluation pipelines

Developers extending MMLU with custom subjects or prompt templates

Requires

MMLU dataset with train/test split per subject

Subject name mapping (e.g., 'abstract_algebra' → 'Abstract Algebra')

Python 3.6+ with string formatting capabilities

Limitations

Fixed 5-example few-shot format — no support for zero-shot or variable-shot evaluation without code modification

Example selection is deterministic (first 5 training examples) — no randomization or stratified sampling

Prompt format is rigid (Question → Options → Answer) — no support for alternative prompt templates or chain-of-thought formatting

What makes it unique

vs alternatives

More reproducible than manual prompt engineering and more standardized than ad-hoc formatting, ensuring that performance differences reflect model capability rather than prompt variation

context-aware prompt truncation via bpe tokenization

Medium confidence

Solves for

Best for

Evaluating models with limited context windows (e.g., older models, edge deployments)

Automated evaluation pipelines that need to handle variable-length prompts robustly

Researchers studying the impact of context length on few-shot learning performance

Requires

Python 3.6+ with tiktoken or equivalent BPE encoder library

Internet access to download encoder resources on first run (cached locally thereafter)

Text input (prompt string)

Limitations

Fixed 2048-token limit — no support for dynamic limits based on model capabilities

BPE tokenization may not align with model's actual tokenizer (e.g., models using SentencePiece or Tiktoken)

Truncation is lossy — removes examples or question context, potentially degrading evaluation validity

What makes it unique

vs alternatives

More robust than character-count-based truncation (which doesn't account for tokenization) and more general than model-specific truncation (which requires per-model configuration)

model calibration measurement across confidence metrics

Medium confidence

Solves for

Best for

ML engineers building production systems where confidence scores drive downstream decisions (e.g., routing to human review)

Researchers studying model reliability and uncertainty quantification

Teams evaluating whether fine-tuning or instruction-tuning improves model calibration

Requires

Model predictions with confidence scores (probabilities for each class)

Ground truth labels for evaluation set

Python 3.6+ with numpy for metric computation

Limitations

Requires model to output confidence scores or probabilities — doesn't work with models that only output discrete answers

Calibration metrics are aggregate statistics — don't identify which specific classes or domains are miscalibrated

Different metrics (ECE, SCE, ACE) can rank models differently — no consensus on which metric is most important

What makes it unique

vs alternatives

hierarchical subject organization and result aggregation

Medium confidence

Solves for

Best for

Researchers analyzing model strengths and weaknesses across knowledge domains

Teams building domain-specific models and wanting to validate breadth of knowledge

Educators or curriculum designers studying how LLMs understand different subjects

Requires

MMLU dataset with subject labels and category mappings

Python 3.6+ with dict/list operations for aggregation

Per-subject accuracy scores from evaluation

Limitations

Subject categorization is fixed — no support for custom hierarchies or alternative taxonomies

Aggregation is simple averaging — doesn't weight subjects by importance or difficulty

57 subjects provide broad coverage but uneven depth (some subjects have 50 questions, others 100+)

What makes it unique

vs alternatives

More structured than flat subject lists and more informative than single overall scores, enabling researchers to identify domain-specific weaknesses and guide targeted model improvements

standardized evaluation harness with reproducible model testing

Medium confidence

Solves for

I want to evaluate my model on MMLU with minimal custom codeI need reproducible evaluation results that match published benchmarksI want to compare multiple models using the same evaluation protocol

Best for

ML engineers and researchers evaluating models without deep benchmark expertise

Teams integrating MMLU evaluation into CI/CD pipelines or automated testing

Researchers publishing results that need to be reproducible and comparable to prior work

Requires

MMLU dataset (hendrycks/test repository)

Model implementation (FLAN or compatible interface)

Python 3.6+ with file I/O and CSV writing capabilities

Limitations

Harness is specific to FLAN models — extending to other model families requires code modification

No built-in support for batch evaluation or distributed evaluation across multiple GPUs/TPUs

Results are written to CSV files — no structured output format (JSON, database) for programmatic access

What makes it unique

vs alternatives

More complete than individual evaluation functions and more reproducible than manual evaluation scripts, enabling consistent benchmarking across teams and time periods

structured subject category taxonomy and hierarchical organization

Medium confidence

Solves for

Best for

Researchers analyzing model performance patterns across knowledge domains

Teams publishing MMLU results with category-level breakdowns

Developers building analysis tools that need subject-to-category mappings

Requires

categories.py file with subject-to-category mappings

Python 3.7+ for importing and using taxonomy

Limitations

Taxonomy is fixed and immutable — cannot add new subjects or reorganize categories without modifying source code

Category definitions are coarse-grained (4 categories) — may obscure fine-grained performance patterns within categories

No weighting or importance ranking — all subjects treated equally in aggregation

What makes it unique

vs alternatives

Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.

Alternatives to MMLU

v087Product

AI UI generator by Vercel — creates production-quality React/Next.js components from natural language descriptions.

Compare →

Framer82Product

AI-powered website design and publishing — generates responsive, professionally designed sites from descriptions.

Compare →

Midjourney79Product

AI image generation — artistic high-quality outputs, Discord bot, photorealistic V6 model.

Compare →

xCodeEval67Benchmark

Multilingual code evaluation across 17 languages.

Compare →

MMLU

Capabilities7 decomposed

few-shot multitask evaluation across 57 knowledge domains

prompt generation with few-shot example formatting

context-aware prompt truncation via bpe tokenization

model calibration measurement across confidence metrics

hierarchical subject organization and result aggregation

standardized evaluation harness with reproducible model testing

structured subject category taxonomy and hierarchical organization

Related Artifactssharing capabilities

Qwen3-8B

MiniMax: MiniMax M2.1

OpenAI: GPT-5.2

Qwen2.5-0.5B-Instruct

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (GPT-NeoX)

Qwen2.5-3B-Instruct

Best For

Known Limitations

Requirements

Input / Output

UnfragileRank

About

Categories

Alternatives to MMLU

Are you the builder of MMLU?

Get the weekly brief

Data Sources

MMLU

Capabilities7 decomposed

few-shot multitask evaluation across 57 knowledge domains

prompt generation with few-shot example formatting

context-aware prompt truncation via bpe tokenization

model calibration measurement across confidence metrics

hierarchical subject organization and result aggregation

standardized evaluation harness with reproducible model testing

structured subject category taxonomy and hierarchical organization

Related Artifactssharing capabilities

Qwen3-8B

MiniMax: MiniMax M2.1

OpenAI: GPT-5.2

Qwen2.5-0.5B-Instruct

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (GPT-NeoX)

Qwen2.5-3B-Instruct

Best For

Known Limitations

Requirements

Input / Output

UnfragileRank

About

Categories

Alternatives to MMLU

Are you the builder of MMLU?

Get the weekly brief

Data Sources