LiveCodeBench

Detects data contamination by annotating each benchmark problem with its release date from competitive programming platforms (LeetCode, AtCoder, Codeforces) and comparing against model training cutoff dates. When a model's performance drops sharply on problems released after its training date, contamination is inferred. This mechanism works by partitioning the benchmark into temporal cohorts and analyzing performance degradation patterns across release windows.

Evaluates code generation models across three distinct scenarios—code generation from specifications, self-repair of broken code, and test output prediction—each testing different cognitive capabilities. The benchmark runs the same model against all three scenarios and produces scenario-specific rankings, revealing that models have inconsistent relative performance (e.g., Claude-3-Opus outperforms GPT-4-turbo on test output prediction but not code generation). This multi-scenario approach prevents single-task benchmark gaming and exposes model specialization patterns.

Partitions the benchmark into difficulty tiers, with an explicitly labeled 'LCB-Easy' subset for easier problems. This enables separate evaluation of model performance on easy vs. hard problems, revealing whether models have consistent capability across difficulty levels or whether they degrade on harder problems. The easy subset is used to detect overfitting in models that perform well on HumanEval but poorly on LCB-Easy, suggesting the models overfit to HumanEval's specific problem distribution rather than learning generalizable code generation skills.

Provides open access to the benchmark dataset (300+ problems with test cases) and reference implementation code via public repositories. This enables researchers and practitioners to run local evaluations, analyze benchmark properties, and build custom evaluation pipelines. The open-source approach promotes transparency, reproducibility, and community contribution to benchmark maintenance and improvement.

Automatically updates the public leaderboard as new problems are added to the benchmark and models are re-evaluated against the expanded problem set. This ensures the leaderboard reflects the current benchmark state and prevents models from achieving artificially high scores on a fixed problem set. The continuous update mechanism is enabled by the automated problem ingestion pipeline and evaluation infrastructure.

Automatically ingests new problems from active competitive programming platforms (LeetCode, AtCoder, Codeforces) on an ongoing basis, with problems dated by their release on the source platform. The benchmark maintains a rolling window of problems (300+ as of documentation) spanning May 2023 to February 2024 and beyond, preventing stagnation and ensuring that new model evaluations always include unseen problems. This continuous refresh is the core mechanism preventing data contamination—models trained before a problem's release date cannot have seen it.

Executes generated code in an isolated sandbox environment against competitive programming test cases with defined inputs and expected outputs. The execution environment enforces timeout and resource limits (specifics unknown) and validates that generated code produces correct output for all test cases. This capability is required for both code generation evaluation (does the code run and produce correct output?) and test output prediction evaluation (does the model correctly predict what the code will output?). The sandbox prevents malicious or resource-exhausting code from affecting the evaluation infrastructure.

Maintains a public leaderboard that ranks models separately for each evaluation scenario (code generation, self-repair, test output prediction) rather than a single aggregate score. The leaderboard is continuously updated as new problems are added to the benchmark and new models are evaluated. Rankings reveal that models have inconsistent relative performance across scenarios—for example, Claude-3-Opus ranks highest on test output prediction but not on code generation, while GPT-4-turbo ranks highest on code generation. This scenario-specific ranking prevents misleading single-metric comparisons and exposes model specialization.

Identifies models that perform well on HumanEval but poorly on LiveCodeBench-Easy by comparing rankings across benchmarks. The analysis reveals that some fine-tuned models (e.g., DS-Ins-1.3B) achieve high HumanEval scores but 'considerably worse' performance on LCB-Easy, suggesting overfitting to HumanEval's specific problem distribution or evaluation methodology. This comparative analysis is enabled by LiveCodeBench's multi-scenario design and real competitive programming problems, which test different capabilities than HumanEval's synthetic problems.

Evaluates both open-source and closed API-access models on the same benchmark and publishes comparative performance data. The analysis reveals that closed API-access models (GPT-4-turbo, Claude-3-Opus) systematically outperform open models, with only fine-tuned variants of 30+B parameter models approaching closed model performance. This comparison enables practitioners to make informed trade-offs between model cost, latency, privacy, and capability when selecting code generation models.

Sources code generation problems from three active competitive programming platforms (LeetCode, AtCoder, Codeforces) and curates them into a benchmark dataset with standardized format. Each problem includes a natural language specification, input/output examples, and test cases with defined constraints. Problems are selected to represent diverse difficulty levels and algorithmic concepts, with a subset labeled as 'LCB-Easy' for easier problems. This sourcing approach ensures problems are real, non-synthetic, and have been validated by thousands of competitive programmers.

Evaluates models' ability to fix broken code by providing a code snippet with an error and asking the model to repair it. The benchmark measures whether the model can identify the error, understand the intended functionality, and generate corrected code that passes test cases. This capability tests a different cognitive skill than code generation from scratch—repair requires understanding existing code structure and intent rather than generating from specifications. The specific task format (e.g., how errors are introduced, whether partial repairs are credited) is not fully documented.

Evaluates models' ability to predict what code will output given test inputs without actually executing the code. The model is provided with a code snippet and test inputs, and must predict the output without running the code. This tests the model's code understanding and reasoning capability—can it trace through code logic and predict results? This scenario is distinct from code generation (does the model write correct code?) and self-repair (can the model fix broken code?). Claude-3-Opus outperforms GPT-4-turbo on this scenario, suggesting different models have different reasoning strengths.