Problem Specific Evaluator Integration And Customization

via “custom evaluation prompt configuration”

Real-world user query benchmark judged by GPT-4.

Unique: Enables users to customize GPT-4 judge prompts for domain-specific evaluation criteria, rather than forcing all evaluations to use fixed helpfulness/safety/instruction-following dimensions. Supports experimentation with different evaluation rubrics and alignment with organizational values.

vs others: More flexible than fixed-criteria benchmarks because it allows domain-specific customization; more practical than building custom evaluation infrastructure because it reuses the WildBench query dataset and judge infrastructure; more transparent than black-box evaluation because users control the evaluation criteria