Vote Aggregation And Statistical Confidence Estimation

Crowdsourced LLM evaluation — side-by-side blind voting, Elo ratings, most trusted LLM benchmark.

Unique: Moves beyond point estimates (Elo scores) to quantify uncertainty in rankings, enabling principled interpretation of benchmark results. Provides confidence intervals that widen when vote volume is low, preventing over-confident claims about model differences.

vs others: More rigorous than raw win-rate leaderboards because it accounts for statistical noise; more transparent than single-point Elo scores because it shows confidence bounds

via “dynamic leaderboard ranking with statistical confidence intervals”

arena-leaderboard — AI demo on HuggingFace

Unique: Combines Elo rating aggregation with Bayesian confidence interval estimation to quantify ranking uncertainty, making statistical reliability explicit rather than hidden. Enables incremental leaderboard updates as votes accumulate while maintaining confidence bounds that reflect data sparsity.

vs others: More statistically rigorous than simple win-rate rankings because confidence intervals account for vote count, and more transparent than fixed-benchmark leaderboards because uncertainty is quantified and displayed.

via “classification accuracy improvement via majority voting aggregation”

* 🏆 1998: [Gradient-based learning applied to document recognition (CNN/GTN)](https://ieeexplore.ieee.org/abstract/document/726791)

Unique: Applies simple plurality voting without confidence weighting or adaptive aggregation, relying on error decorrelation from bootstrap resampling to achieve accuracy gains — a theoretically grounded approach that contrasts with weighted voting schemes by treating all ensemble members equally and depending entirely on bootstrap-induced diversity

vs others: Simpler than weighted voting or stacking (no meta-learner required) and more interpretable than neural network ensembles, but less adaptive than boosting-based methods that explicitly weight classifiers by accuracy