Extensible Task System For Adding New Evaluation Scenarios

Embedding model benchmark — 8 tasks, 112 languages, the standard for comparing embeddings.

Unique: AbsTask base class defines a minimal interface (load_data, evaluate) that subclasses override to implement task-specific logic. Task registry enables dynamic task discovery and selection. Task metadata (language, domain, license) is standardized and used for filtering. This design separates task logic from evaluation orchestration, enabling new tasks to be added without modifying core code.

vs others: Extensible task framework vs. monolithic evaluation code, enabling new tasks to be added without modifying core logic. Task registry enables dynamic task discovery vs. static task lists.