Elo Rating Computation For Model Ranking

via “elo-rating-computation-for-model-ranking”

Crowdsourced Elo ratings from human model comparisons.

Unique: Applies chess-style Elo rating system to LLM evaluation, enabling dynamic ranking updates as new preference data arrives and providing a single comparable metric across all models without requiring predefined performance thresholds or absolute scoring rubrics

vs others: Simpler and more transparent than learned preference models while capturing preference dynamics better than static win-rate metrics, though less interpretable than absolute performance scores and vulnerable to saturation when models are similar in quality