Open LLM Leaderboard

Automatically evaluates open-source LLMs against a fixed suite of standardized benchmarks (MMLU, HellaSwag, ARC, TruthfulQA, Winogrande, GSM8K) using a unified evaluation harness. The pipeline ingests model weights from Hugging Face Hub, runs inference on each benchmark with consistent prompting and sampling strategies, and aggregates results into normalized scores. Uses vLLM or similar inference optimization for efficient batch evaluation across diverse model architectures.

Maintains a live-updating leaderboard that ranks models by aggregate benchmark performance, with version history and submission timestamps. The system tracks when models were evaluated, allows filtering by model size/architecture/license, and displays trend data showing how model performance has evolved. Built as a Hugging Face Space using Gradio for the UI, with backend evaluation jobs queued and executed asynchronously, storing results in a persistent database indexed by model ID and evaluation timestamp.

Provides a submission interface where model developers can register their models for evaluation by providing a Hugging Face model card URL. The system validates the model is publicly accessible, queues it for evaluation against the standard benchmark suite, and notifies the submitter when results are available. Uses a job queue (likely Celery or similar) to manage evaluation tasks, with priority handling for popular models and rate limiting to prevent infrastructure overload. Evaluation jobs are containerized and run in isolated environments to prevent interference between model evaluations.

Disaggregates overall model performance into per-benchmark scores (MMLU, HellaSwag, ARC, TruthfulQA, Winogrande, GSM8K), allowing users to filter and sort models by performance on specific tasks. The UI displays a matrix view where rows are models and columns are benchmarks, with color-coded cells indicating relative performance. Users can click into individual benchmarks to see detailed metrics (accuracy, F1, etc.) and compare models on specific capability dimensions (knowledge, reasoning, common sense).

Displays comprehensive metadata for each evaluated model including architecture, training data, license, parameter count, quantization status, and evaluation methodology. The leaderboard links to model cards, papers, and GitHub repositories, and documents the exact prompts, sampling parameters, and benchmark versions used in evaluation. This enables reproducibility — users can understand exactly how scores were computed and potentially replicate evaluations locally. Metadata is extracted from Hugging Face model cards and supplemented with manual curation for popular models.

Allows users to filter models by parameter count, quantization level, and estimated memory requirements, enabling selection of models that fit within computational constraints. The leaderboard displays model size metadata and provides filtering controls to show only models below a specified size threshold. This helps users find the best-performing model that can run on their available hardware (e.g., 'best model under 7B parameters', 'best quantized model under 8GB VRAM'). Size information is extracted from model cards and supplemented with inference benchmarks.

Displays license information for each model (MIT, Apache 2.0, OpenRAIL, commercial restrictions, etc.) and provides filtering to show only models with specific license types. The leaderboard aggregates license data from Hugging Face model cards and highlights models with permissive vs restrictive licenses. This enables teams to filter for models that meet their legal and compliance requirements without manual license checking.

Displays model architecture information (Transformer, MoE, RNN, etc.) and framework compatibility (PyTorch, TensorFlow, ONNX, etc.) for each model. Users can filter by architecture or framework to find models compatible with their deployment infrastructure. This metadata is extracted from model cards and supplemented with inference framework testing results.

Enables users to select multiple models and view their performance side-by-side across all benchmarks, with visual comparison charts and difference calculations. The comparison view shows absolute scores, relative performance differences, and highlights areas where models diverge significantly. This is implemented as an interactive UI feature allowing users to add/remove models from comparison and customize visualization (bar charts, radar charts, tables).

Documents the exact evaluation methodology including benchmark versions, prompt templates, sampling parameters (temperature, top-p, max tokens), and inference framework used. This information is displayed alongside results and made available for download, enabling users to replicate evaluations locally or understand potential sources of variance. The leaderboard maintains version history of evaluation methodology, allowing users to understand how methodology changes have affected scores over time.