Difficulty Stratified Problem Categorization And Filtering

via “problem-difficulty-and-category-stratification”

Continuously updated coding benchmark — new competitive programming problems, prevents contamination.

Unique: Enables stratified analysis of model performance across difficulty levels and problem categories, revealing whether models have consistent capability or show degradation on harder problems. This level of detail is not provided by single-metric benchmarks.

vs others: More granular than aggregate leaderboards because it enables analysis of performance across problem subsets, revealing capability gaps that aggregate metrics might hide.

via “difficulty-stratified problem categorization and filtering”

10K coding problems across 3 difficulty levels with test suites.

Unique: Explicitly stratifies problems into three difficulty tiers with substantial size per tier (3.6K, 5K, 1.4K), enabling fine-grained analysis of model performance degradation across skill levels rather than treating all problems as equal difficulty

vs others: Unlike HumanEval which lacks difficulty stratification, APPS enables researchers to measure whether models have genuine reasoning or are pattern-matching, by comparing performance across tiers