Extensible Task System For Adding New Evaluation Scenarios

Embedding model benchmark — 8 tasks, 112 languages, the standard for comparing embeddings.

Unique: AbsTask base class defines a minimal interface (load_data, evaluate) that subclasses override to implement task-specific logic. Task registry enables dynamic task discovery and selection. Task metadata (language, domain, license) is standardized and used for filtering. This design separates task logic from evaluation orchestration, enabling new tasks to be added without modifying core code.

vs others: Extensible task framework vs. monolithic evaluation code, enabling new tasks to be added without modifying core logic. Task registry enables dynamic task discovery vs. static task lists.

via “scenario-based evaluation harness with standardized datasets and metrics”

Stanford's holistic LLM evaluation — 42 scenarios, 7 metrics including fairness, bias, toxicity.

Unique: Implements scenarios as first-class objects with encapsulated datasets, prompts, and metrics, allowing each scenario to define its own success criteria and evaluation methodology. Uses public, versioned datasets to ensure reproducibility across time and teams.

vs others: More modular and extensible than monolithic evaluation scripts because each scenario is self-contained, enabling easy addition of new scenarios or modification of existing ones without affecting others