Custom Task Definition Via Python Classes With Metric Registration

via “task-specific metric computation and result aggregation”

Embedding model benchmark — 8 tasks, 112 languages, the standard for comparing embeddings.

Unique: Task-specific evaluators inherit from a base evaluator class and implement compute() methods that handle metric calculation for each task type. Metrics are computed in-memory with caching to avoid redundant computation. Results are aggregated using a standardized format (JSON) that preserves per-task breakdowns and enables post-hoc analysis. This design separates metric logic from evaluation orchestration.

vs others: Task-specific evaluators vs. generic metric libraries (e.g., scikit-learn) ensure metrics are computed correctly for each task type. Standardized result format enables leaderboard integration and reproducible comparisons.

EleutherAI's evaluation framework — 200+ benchmarks, powers Open LLM Leaderboard.

Unique: Provides a Task base class that users can extend to implement custom evaluation logic, with automatic registration in the global task registry. Custom tasks can override request generation, metric computation, and result aggregation. Metrics are registered separately and can be reused across tasks, enabling modular metric development.

vs others: Enables arbitrary Python logic for task definition and metrics, whereas YAML-based tasks are limited to built-in capabilities; integrates custom tasks into the evaluation pipeline with automatic batching and caching support