HumanEval
BenchmarkFreeOpenAI's code generation benchmark — 164 Python problems with unit tests, pass@k evaluation.
Capabilities8 decomposed
hand-crafted programming problem dataset with canonical solutions
Medium confidenceProvides a curated collection of 164 hand-written Python programming problems, each with a function signature prompt, canonical reference implementation, and comprehensive test cases. Problems are stored in JSONL.gz format and loaded via the read_problems() function, enabling standardized evaluation of code generation models across diverse algorithmic and implementation challenges.
Hand-crafted by OpenAI researchers specifically for code generation evaluation, not auto-generated or scraped from existing sources. Each problem includes a canonical solution and carefully designed test cases that verify functional correctness rather than just syntax.
More authoritative and widely-adopted than alternatives like MBPP or CodeXGLUE because it was created by OpenAI and has become the de facto standard for publishing code generation results, enabling direct comparison across papers and models.
sandboxed code execution with timeout and resource limits
Medium confidenceExecutes untrusted generated code in an isolated environment via the unsafe_execute() function, which applies timeout constraints and resource monitoring to prevent infinite loops, memory exhaustion, and system resource abuse. The execution engine wraps code in a try-except block and captures stdout/stderr, enabling safe evaluation of arbitrary code without compromising the host system.
Implements a lightweight sandbox using Python subprocess isolation with explicit timeout handling and exception capture, rather than relying on heavy containerization. This makes it fast and portable while still preventing the most common failure modes (infinite loops, crashes).
Faster and simpler to deploy than Docker-based sandboxing used by some alternatives, while still providing adequate safety for research evaluation; trade-off is weaker isolation guarantees compared to OS-level sandboxing.
functional correctness testing via unit test execution
Medium confidenceTests generated code against problem-specific test cases via the check_correctness() function, which executes the generated function with each test input and compares output against expected results. Test cases are embedded in the problem definition and executed sequentially, with the function marked as correct only if all tests pass without exceptions or timeouts.
Integrates test execution directly into the evaluation pipeline rather than as a separate step, allowing tight coupling between problem definition and test harness. Tests are embedded in the problem JSONL and executed in the same sandboxed environment as the generated code.
More integrated and standardized than ad-hoc testing approaches; provides consistent test execution semantics across all 164 problems, whereas custom test harnesses may have subtle differences in how they invoke and validate code.
pass@k metric calculation with unbiased statistical estimation
Medium confidenceCalculates the pass@k metric via estimate_pass_at_k(), which estimates the probability that at least one of k code samples passes all tests, using an unbiased estimator that accounts for sampling variance. The function takes the number of problems, number of samples per problem, and number of passing samples, then computes the pass@k statistic with confidence intervals, enabling fair comparison across models that generate different numbers of candidates.
Implements an unbiased estimator for pass@k that corrects for sampling bias, rather than using naive pass rates. The estimator accounts for the probability that at least one sample passes, using combinatorial statistics to avoid overestimating performance when k is large relative to the number of samples.
More statistically rigorous than simple pass rate calculations; enables fair comparison between models that generate 1 sample vs 100 samples, whereas naive metrics would penalize models that generate fewer candidates even if they're higher quality.
jsonl-based completion format parsing and result serialization
Medium confidenceHandles reading code completions from JSONL files via stream_jsonl() and writing evaluation results via write_jsonl(), supporting a standardized format where each line is a JSON object containing task_id, completion, and optional metadata. This enables integration with external code generation pipelines that output completions in JSONL format, and allows downstream analysis tools to consume evaluation results in the same structured format.
Standardizes the input/output format for code generation evaluation, allowing any model or pipeline to generate completions in JSONL format and feed them into HumanEval without custom adapters. The format is simple enough to be language-agnostic while structured enough to preserve metadata.
More flexible than alternatives that require specific API calls or Python object formats; JSONL is language-agnostic and can be generated by any code generation system, making HumanEval accessible to researchers using non-Python frameworks.
command-line evaluation workflow orchestration
Medium confidenceProvides a CLI tool (evaluate_functional_correctness) that orchestrates the full evaluation pipeline: reading completions from JSONL, executing tests via check_correctness(), calculating pass@k metrics via estimate_pass_at_k(), and writing results to output JSONL. The CLI accepts parameters like k values and input file path, handling the entire workflow without requiring Python scripting.
Provides a single entry point that chains together data loading, code execution, metric calculation, and result serialization, eliminating the need for users to write orchestration code. The CLI is installed as a setuptools entry point, making it available as a system command after package installation.
More accessible than requiring users to write Python code to import and call individual functions; the CLI makes HumanEval usable by non-Python developers and integrates naturally into shell-based workflows and CI/CD systems.
problem-specific test case validation with entry point routing
Medium confidenceRoutes code execution to the correct function entry point specified in each problem definition, enabling evaluation of generated code that may define multiple functions or classes. The entry_point field in each problem specifies which function to call during testing, and the execution engine uses this to invoke the correct callable, supporting problems where the generated code must define helper functions or classes alongside the main solution.
Decouples the entry point from the function signature, allowing problems to specify which callable to test even if the generated code defines multiple functions. This is stored as metadata in the problem definition rather than inferred from the code, providing explicit control over which function is tested.
More flexible than alternatives that assume the entry point is always the first or only function defined; explicit entry point specification enables testing of code with helper functions or multiple implementations without ambiguity.
exception and timeout error reporting with execution diagnostics
Medium confidenceCaptures and reports execution failures including timeouts, exceptions, and assertion errors via the check_correctness() function, which wraps test execution in try-except blocks and returns detailed error information. The system distinguishes between different failure modes (timeout, exception, assertion failure) and includes the exception message or traceback, enabling diagnosis of why generated code failed.
Provides structured error reporting that distinguishes between different failure modes (timeout vs exception vs assertion), rather than treating all failures as identical. This enables analysis of whether models tend to produce code that hangs, crashes, or produces wrong answers.
More informative than simple pass/fail reporting; the detailed error information enables root cause analysis of model failures, whereas alternatives that only report pass/fail provide no insight into why code failed.
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with HumanEval, ranked by overlap. Discovered automatically through the match graph.
phantom-lens
A Cluely / Interview Coder alternative with features we probably shouldn’t talk about, built for winning exams..
CodeContests
13K competitive programming problems from AlphaCode research.
LiveCodeBench
Continuously updated coding benchmark — new competitive programming problems, prevents contamination.
Code Coach
Master FAANG interviews with AI-driven, instant-feedback...
MBPP+
Enhanced Python coding benchmark with rigorous testing.
MBPP (Mostly Basic Python Problems)
974 basic Python problems complementing HumanEval for code evaluation.
Best For
- ✓ML researchers evaluating code generation models
- ✓teams building LLM-based coding assistants
- ✓academic institutions studying code synthesis
- ✓researchers evaluating code generation models in untrusted environments
- ✓CI/CD pipelines that need to test LLM-generated code automatically
- ✓teams building code generation APIs that accept user-submitted models
- ✓evaluating code generation models on well-defined algorithmic problems
- ✓automated testing of LLM-generated code in CI/CD pipelines
Known Limitations
- ⚠Limited to 164 problems — may not capture domain-specific coding patterns in specialized fields
- ⚠Python-only — cannot evaluate code generation for other languages directly
- ⚠Hand-crafted problems may have implicit biases toward certain algorithmic styles
- ⚠No problem difficulty stratification — mix of easy and hard problems without explicit categorization
- ⚠Timeout mechanism is process-level, not instruction-level — may not catch tight infinite loops quickly
- ⚠No true sandboxing on non-Unix systems — relies on Python's subprocess isolation which is weaker than OS-level sandboxing
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
OpenAI's benchmark for evaluating code generation. 164 hand-crafted Python programming problems with unit tests. Measures functional correctness (pass@k). The original and most cited code generation benchmark.
Categories
Alternatives to HumanEval
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Compare →Amplication brings order to the chaos of large-scale software development by creating Golden Paths for developers - streamlined workflows that drive consistency, enable high-quality code practices, simplify onboarding, and accelerate standardized delivery across teams.
Compare →Are you the builder of HumanEval?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →