OSWorld

Evaluates multimodal AI agents by executing open-ended computer tasks on actual operating systems (Ubuntu, Windows, macOS) with real applications, file systems, and GUI interactions. Uses custom execution-based evaluation scripts per task that verify task completion against initial state setup configurations, enabling reproducible assessment of agent performance on authentic desktop workflows without simulation or constraint.

Provides unified task execution framework across Ubuntu, Windows, and macOS by standardizing task definitions, initial state setup, and evaluation scripts for each OS variant. Abstracts OS-specific differences in file paths, application availability, and GUI rendering while maintaining task semantic equivalence, allowing single benchmark to assess agent generalization across heterogeneous desktop environments.

Maintains comprehensive documentation including research paper, code repository, slides, and community channels (Discord, Twitter) for benchmark usage, contribution, and discussion. Provides multiple formats for learning benchmark methodology and engaging with research community.

Evaluates agent capability to visually ground interface elements from screenshots and understand GUI layouts, button positions, text fields, and application state. Agents receive screenshot images as input and must interpret visual information to determine next actions, testing multimodal understanding of desktop interfaces without explicit element annotations or accessibility trees.

Defines and evaluates complex tasks requiring agents to coordinate actions across multiple applications (web browsers, file managers, text editors, desktop apps) within single workflow. Tasks test agent ability to maintain context across application switches, transfer data between apps, and sequence operations across heterogeneous tools to accomplish higher-level goals.

Evaluates agent capability to perform file system operations including file creation, deletion, copying, moving, renaming, and directory navigation. Tests agent understanding of file paths, directory hierarchies, file permissions, and ability to locate files by name or content, verifying task completion through file system state inspection.

Provides per-task initial state configurations and custom evaluation scripts that verify task completion deterministically. Each task includes detailed setup instructions to establish consistent initial OS state and evaluation logic that checks whether agent actions resulted in desired outcome, enabling reproducible benchmarking without human judgment or manual verification.

Establishes human baseline by having human evaluators complete benchmark tasks on the same OS environments and measuring success rate (72.36% documented). Provides ground truth for agent performance comparison and identifies task difficulty ceiling, enabling assessment of whether agent performance gaps reflect task difficulty or model limitations.

Maintains curated dataset of 369 tasks derived from real-world computer use cases with version control and community feedback integration. Recent OSWorld-Verified release (2025-07-28) incorporates community-reported fixes and improvements, enabling continuous benchmark quality enhancement while maintaining historical comparability through versioning.

Provides AWS integration for distributed task execution reducing full benchmark evaluation time to approximately 1 hour (as of 2025-07-28 upgrade). Abstracts infrastructure complexity and enables researchers without local multi-OS environments to evaluate agents at scale, though self-hosted evaluation on local Ubuntu/Windows/macOS remains supported.

Provides web-based data viewer for exploring benchmark tasks, initial states, evaluation scripts, and results interactively. Enables researchers to inspect individual task definitions, understand evaluation methodology, and analyze failure patterns without downloading full dataset or running evaluation infrastructure.