RealWorldQA

Evaluates multimodal models' ability to understand spatial relationships, object positioning, and geometric reasoning in natural photographs. The benchmark presents images with spatial queries (e.g., 'What is to the left of the person?', 'How many objects are between X and Y?') and measures whether models can correctly interpret 2D spatial layouts, occlusion, depth cues, and relative positioning without synthetic or annotated spatial metadata.

Measures multimodal models' ability to accurately count and quantify objects in real-world images through questions like 'How many people are in the image?' or 'Count the number of cars visible.' The benchmark evaluates both exact counting accuracy and approximate quantification, testing whether models can enumerate objects despite occlusion, varying scales, and visual clutter typical of natural photographs.

Evaluates multimodal models' ability to read and extract text from real-world images, including signs, labels, documents, and text in natural scenes. The benchmark presents images containing visible text and asks models to read, transcribe, or answer questions about the text content, testing optical character recognition (OCR) capabilities integrated into vision-language models without explicit OCR preprocessing.

Evaluates multimodal models' ability to apply common-sense knowledge and reasoning to answer questions about real-world images that require world knowledge beyond pure visual analysis. Questions may ask about object purposes, likely actions, social context, or practical implications (e.g., 'Why would someone use this tool?' or 'What is this person likely doing?'). The benchmark tests integration of visual understanding with semantic reasoning and knowledge about real-world conventions.

Provides a standardized evaluation framework for comparing performance across different vision-language models on a consistent set of real-world image questions. The benchmark infrastructure supports loading model outputs, computing accuracy metrics (exact match, semantic similarity), and generating comparative performance reports across models and question categories (spatial, counting, text, reasoning).

Curates a collection of real-world photographs with manually annotated question-answer pairs covering spatial reasoning, counting, text reading, and common-sense understanding. The dataset construction involves image selection from diverse real-world scenarios, question generation by human annotators, and answer validation to ensure quality and diversity of reasoning types, creating a resource for training and evaluating multimodal models on practical visual understanding tasks.