Custom Execution Based Task Evaluation

via “custom execution-based task evaluation”

Real OS benchmark for multimodal computer agents.

Unique: Uses custom per-task evaluation scripts rather than generic scoring functions, enabling task-specific success criteria that capture domain knowledge (e.g., correct file format, application-specific state changes). This approach is more accurate than generic metrics but requires significant engineering effort and domain expertise per task.

vs others: More accurate than generic scoring functions for complex, multi-step tasks, but less scalable and harder to maintain than standardized evaluation metrics used in simpler benchmarks.