Program Synthesis Task Generation And Evaluation With Pass K Metrics

via “comprehensive test suite execution and pass-rate evaluation”

10K coding problems across 3 difficulty levels with test suites.

Unique: Provides 21 test cases per problem on average (vs single example in HumanEval), enabling rigorous pass-rate evaluation and pass@k metrics that measure robustness across multiple test cases rather than single-shot correctness

vs others: Comprehensive test suites catch partial solutions and edge case failures that single-example evaluation would miss, providing more reliable quality signals for code generation systems