MBPP (Mostly Basic Python Problems)

Provides a curated dataset of 974 Python programming problems with reference implementations and test cases to systematically evaluate code generation models. Each problem includes a natural language task description, a correct solution function, and three validation test cases that can be executed to measure pass/fail rates. The dataset is structured as Hugging Face Dataset objects enabling direct integration with model evaluation pipelines via the datasets library.

Executes generated Python code against reference test cases and computes aggregate pass rates. The capability runs each generated solution function with the three provided test inputs, captures execution results (pass/fail/error), and aggregates metrics across the full 974-problem dataset. Integration with Python's exec() or subprocess execution enables safe evaluation of untrusted generated code with timeout and resource limits.

Organizes the 974 problems into semantic categories covering fundamental programming concepts: string manipulation, list/array operations, mathematical functions, sorting/searching, data structure algorithms, and control flow. Each problem is tagged with its primary concept(s), enabling analysis of model performance by programming domain. This taxonomy allows researchers to identify capability gaps — e.g., 'model passes 90% of string problems but only 40% of sorting problems' — and correlate performance with training data composition.

Enables side-by-side evaluation of multiple code generation models (GPT-4, Claude, Copilot, open-source LLMs) on the same 974 problems with consistent test execution. The framework standardizes input/output formats, test case execution, and metric calculation across models with different APIs and output formats. Results are aggregated into comparison matrices showing per-model pass rates, per-problem winner, and statistical significance tests.

Provides problem descriptions in a structured, language-agnostic format (task description + function signature + test cases) that can be adapted to different prompt templates and model conventions. The core problem representation is decoupled from prompt engineering, allowing researchers to test how different prompting strategies affect model performance on identical problems. This enables controlled experiments varying prompt style, few-shot examples, or chain-of-thought guidance while holding the underlying problem constant.

Maintains versioned snapshots of the 974-problem dataset on Hugging Face Hub with immutable problem definitions, test cases, and reference solutions. Each version is tagged with a release date and can be pinned in evaluation scripts, ensuring that benchmark results remain reproducible across time and teams. The dataset includes metadata (problem ID, creation date, category tags) enabling researchers to cite specific versions in papers and track which version was used in published results.

Natively integrates with Hugging Face's datasets library, model hub, and evaluation frameworks (e.g., evaluate library) through standard interfaces. Problems and test cases are accessible via the datasets.load_dataset() API, enabling one-line integration into evaluation pipelines. The dataset follows Hugging Face conventions for splits, features, and metadata, allowing seamless composition with other benchmarks and evaluation tools in the ecosystem.

Includes a correct reference implementation and three test cases for each of the 974 problems, enabling both positive and negative evaluation modes. The reference solutions are hand-written Python functions demonstrating the expected behavior, while test cases cover typical inputs, edge cases, and boundary conditions. This allows evaluation of generated code by comparing outputs to reference solutions or by running test cases directly, supporting both execution-based and semantic-based evaluation approaches.