Scene Understanding And Spatial Reasoning

via “spatial reasoning and visualization evaluation”

23 hardest BIG-Bench tasks where models initially failed.

Unique: Isolates spatial reasoning as a distinct capability by presenting spatial problems in text form with few-shot examples, testing whether models can build and manipulate mental spatial models without visual input. This approach measures pure spatial reasoning capability.

vs others: More focused on spatial reasoning than general reasoning benchmarks; more challenging than visual spatial reasoning because it requires models to construct spatial models from text descriptions rather than perceiving visual images.