Dynamic Leaderboard Ranking With Statistical Confidence Intervals

via “vote aggregation and statistical confidence estimation”

Crowdsourced LLM evaluation — side-by-side blind voting, Elo ratings, most trusted LLM benchmark.

Unique: Moves beyond point estimates (Elo scores) to quantify uncertainty in rankings, enabling principled interpretation of benchmark results. Provides confidence intervals that widen when vote volume is low, preventing over-confident claims about model differences.

vs others: More rigorous than raw win-rate leaderboards because it accounts for statistical noise; more transparent than single-point Elo scores because it shows confidence bounds