Problem Difficulty And Category Stratification

via “problem difficulty level annotation and stratification”

12.5K competition math problems — AMC/AIME/Olympiad level, 7 subjects, standard math benchmark.

Unique: Provides difficulty stratification based on official competition sources (AMC 10 is easier than AIME which is harder than Olympiad), enabling researchers to analyze whether models scale their reasoning capabilities with problem difficulty. This reveals whether models have robust reasoning or merely memorized easy problem patterns.

vs others: More principled than arbitrary difficulty scoring because it leverages established competition hierarchies, but less precise than learned difficulty metrics based on empirical model performance data.

via “issue difficulty classification and stratification”

AI coding agent benchmark — real GitHub issues, end-to-end evaluation, the standard for code agents.

Unique: Automatically classifies instance difficulty based on objective metrics (lines changed, files modified) rather than manual annotation, enabling scalable stratification without human effort. This allows analysis of agent performance across difficulty levels without requiring subjective difficulty labels.

vs others: More scalable than manual difficulty annotation because it uses objective metrics, and more nuanced than single aggregate metrics because it reveals how agent performance varies with problem complexity.

via “problem-difficulty-and-category-stratification”

Continuously updated coding benchmark — new competitive programming problems, prevents contamination.

Unique: Enables stratified analysis of model performance across difficulty levels and problem categories, revealing whether models have consistent capability or show degradation on harder problems. This level of detail is not provided by single-metric benchmarks.

vs others: More granular than aggregate leaderboards because it enables analysis of performance across problem subsets, revealing capability gaps that aggregate metrics might hide.

via “difficulty-stratified performance analysis”

57-subject benchmark, the standard metric for comparing LLMs.

Unique: Explicitly tags questions with difficulty levels derived from real academic curricula (elementary through professional certification), enabling builders to measure reasoning depth rather than just aggregate knowledge. Most benchmarks report a single score; MMLU's stratification reveals whether improvements are broad or concentrated in easy questions.

vs others: Provides finer-grained difficulty analysis than GSM8K (math-only) or TruthfulQA (single-domain), and the difficulty labels are grounded in real educational standards rather than arbitrary heuristics.

via “multi-domain reasoning task stratification”

23 hardest BIG-Bench tasks where models initially failed.

Unique: Explicitly stratifies tasks by reasoning modality (algorithmic, arithmetic, logical, causal, spatial) rather than treating all hard tasks as monolithic, enabling domain-specific capability assessment. This structure allows researchers to correlate model architecture choices with specific reasoning strengths.

vs others: More analytically useful than generic hard task collections because stratification enables root-cause analysis of reasoning failures; more focused than full BIG-Bench which lacks explicit domain organization.

via “difficulty-calibrated-problem-stratification”

13K competitive programming problems from AlphaCode research.

Unique: Uses empirical runtime metrics (median and 95th percentile from real submissions) to calibrate difficulty rather than subjective classification or problem setter ratings. This grounds difficulty in measurable performance data and enables reproducible difficulty-based dataset splits.

vs others: More objective than subjective difficulty labels (e.g., 'hard' vs 'medium') and more granular than binary easy/hard splits, enabling fine-grained curriculum learning studies that other datasets don't support.

via “diverse conversation category stratification”

183K multi-turn preference comparisons for alignment.

Unique: Explicitly stratifies 183K comparisons across diverse conversation categories rather than treating preference data as a monolithic pool, enabling analysis of how model preferences vary by task type and supporting category-aware training strategies.

vs others: Provides better coverage of diverse conversation types than single-domain preference datasets, enabling more robust general-purpose alignment compared to category-specific datasets that may overfit to narrow use cases

via “difficulty-stratified problem categorization and filtering”

10K coding problems across 3 difficulty levels with test suites.

Unique: Explicitly stratifies problems into three difficulty tiers with substantial size per tier (3.6K, 5K, 1.4K), enabling fine-grained analysis of model performance degradation across skill levels rather than treating all problems as equal difficulty

vs others: Unlike HumanEval which lacks difficulty stratification, APPS enables researchers to measure whether models have genuine reasoning or are pattern-matching, by comparing performance across tiers

via “difficulty-stratified problem sampling and filtering”

12.5K competition math problems across 7 subjects and 5 difficulty levels.

Unique: Pre-assigned difficulty metadata (1-5 scale) from competition context enables efficient filtering without re-evaluation, unlike datasets where difficulty must be computed post-hoc. Difficulty labels are grounded in actual competition difficulty (AMC problems are easier, AIME problems are harder), providing meaningful stratification.

vs others: More efficient than datasets requiring dynamic difficulty estimation because filtering is O(1) lookup on metadata; more reliable than model-specific difficulty metrics because it uses competition-grounded labels that generalize across model architectures.

via “problem categorization and concept mapping”

974 basic Python problems complementing HumanEval for code evaluation.

Unique: Curated categorization by Google Research based on fundamental programming concepts (string, list, math, data structures) rather than algorithmic complexity or problem domain, providing a practical lens for understanding basic coding proficiency across different skill areas

vs others: More granular than treating all problems as a single pool; simpler and more interpretable than complexity-based rankings; directly maps to programming education curricula, making results actionable for model improvement

via “difficulty-based problem retrieval”

Search solved.ac problems by difficulty, tags, and keywords to find the right challenges. Check user ratings, tiers, and solved counts to track progress. Convert natural language into precise filters for faster discovery.

Unique: Integrates a tiered indexing system that allows for rapid retrieval of problems based on difficulty, unlike simpler keyword-based searches.

vs others: Faster and more efficient than traditional databases that do not categorize problems by difficulty.