LiveBench

Maintains a benchmark dataset that automatically incorporates new questions sourced from recent information (news, research, current events) while preventing data leakage through continuous rotation and versioning. Uses a pipeline that ingests fresh content, generates novel evaluation questions, and retires older questions to ensure models cannot have seen test data in their training corpora, addressing the fundamental problem of benchmark contamination in rapidly-evolving LLM evaluation.

Evaluates LLM performance across five distinct capability domains through domain-specific question sets and evaluation metrics. Each domain uses tailored question generation and grading logic: math uses symbolic verification, coding uses execution-based testing, reasoning uses logical consistency checking, language uses semantic similarity metrics, and data analysis uses SQL/pandas execution validation. Questions are sampled from live information sources to ensure domain-specific relevance and novelty.

Grades coding and data analysis responses by actually executing generated code in isolated sandboxed environments rather than string matching or regex validation. For coding tasks, runs generated code against test cases and validates output correctness. For data analysis, executes SQL or pandas code against test datasets and verifies result accuracy. Uses containerization or process isolation to prevent malicious code execution while enabling deterministic evaluation of functional correctness.

Continuously ingests fresh content from recent information sources (news APIs, research databases, current events feeds) and uses this content to generate novel benchmark questions. Implements a pipeline that filters for relevant content, extracts factual claims and scenarios, generates questions with varying difficulty levels, and validates that questions are solvable and non-trivial. This ensures benchmark questions reflect current knowledge and cannot have been in model training data.

Tracks question creation dates, model training cutoffs, and information source publication dates to detect potential data leakage. Implements versioning system where each benchmark snapshot is timestamped and linked to source information, enabling post-hoc analysis of whether a model could have seen a question during training. Uses statistical analysis to identify suspiciously high performance on questions from before training cutoff, flagging potential contamination in model training data.

Maintains a public leaderboard that ranks models by benchmark performance while accounting for contamination risk. Scores are adjusted based on temporal alignment between question sources and model training dates, with lower scores for models evaluated on potentially contaminated questions. Implements filtering to show only 'clean' evaluations where question sources clearly post-date training cutoffs, and provides transparency about contamination risk for each model-benchmark pair.

Evaluates language generation tasks (translation, summarization, paraphrase) by computing semantic similarity between model outputs and reference answers using pre-trained embedding models. Rather than exact string matching, compares vector representations to measure whether generated text captures the same meaning, allowing for multiple valid phrasings. Uses cosine similarity thresholds calibrated to human judgment to determine correctness, with optional human review for borderline cases.

Maintains multiple timestamped versions of the benchmark as questions are added and retired, enabling historical comparison of model performance across benchmark snapshots. Tracks which questions were active in each version, allowing researchers to measure performance on the same question set over time or analyze how model capabilities have changed as the benchmark evolves. Provides APIs to access historical versions and compare results across time periods.