ARC-AGI

Renders abstract reasoning puzzle environments as interactive step-based simulations accessible via Python SDK or REST API, supporting dual render modes: 'terminal' for visual output and headless for high-speed evaluation at 2000+ FPS. Agents interact through GameAction command enums, receiving state updates after each step, enabling real-time agent-environment interaction loops without network latency in local mode.

Measures AI system performance on novel visual puzzles requiring fluid intelligence and abstract reasoning — specifically the ability to recognize patterns in limited examples and generalize to unseen puzzle variants. Tasks are designed to be 'easy for humans, hard for AI' by requiring exploration, perception-to-plan-to-action loops, memory, and goal acquisition without explicit task specifications, forcing genuine reasoning rather than pattern matching on known problem types.

Aggregates agent performance across multiple puzzle tasks into a unified scorecard data structure accessible via `arc.get_scorecard()` method, enabling comparative evaluation of different reasoning systems on the same benchmark. Scorecard system abstracts the underlying scoring formula (pass@k, accuracy, or custom metric) and provides structured output for leaderboard ranking and progress tracking.

Provides three sequential benchmark versions (ARC-AGI, ARC-AGI-2, ARC-AGI-3) representing evolution from static visual puzzles to interactive agent environments, allowing researchers to track progress across versions and identify capability inflection points. Version progression reflects increasing complexity: from pattern recognition to agentic reasoning with memory and goal acquisition.

Provides dual access patterns to benchmark evaluation: Python SDK (`arc_agi.Arcade()`) for local, low-latency evaluation and REST API for remote evaluation and leaderboard submission. SDK supports both authenticated (via ARC_API_KEY) and anonymous access, with authenticated keys enabling 'access to public games at release' and anonymous access providing reduced functionality. REST API enables integration into CI/CD pipelines and cloud-based evaluation infrastructure.

Distributes benchmark as open-source Python toolkit with reference agent implementations and templates, enabling researchers to build custom reasoning systems by extending provided base classes. Toolkit includes game environment abstraction, action enums, and scorecard computation, reducing boilerplate for agent development while maintaining compatibility with official leaderboard evaluation.

Structures ARC Prize 2026 ($2,000,000 total) with explicit requirement that winning solutions be open-sourced, creating financial incentive for public release of novel reasoning techniques. Prize pool distributed across multiple tiers and submission windows via Kaggle partnership, enabling both individual researchers and teams to compete while ensuring breakthrough techniques become public knowledge.

Benchmark tasks are explicitly designed to be 'easy for humans, hard for AI' through human calibration methodology, ensuring evaluation measures genuine reasoning gaps rather than domain-specific knowledge or pattern matching. Tasks require exploration, perception-to-action loops, memory, and goal acquisition — capabilities that humans naturally possess but AI systems struggle with, creating a benchmark resistant to scaling-only approaches.

Evaluates AI systems through multi-step agentic reasoning loops where agents must explore puzzle environments, perceive state changes, plan actions, and execute them iteratively — measuring not just final answers but the reasoning process itself. Interactive format forces agents to maintain memory across steps, acquire goals dynamically, and adapt strategies based on environmental feedback, preventing single-pass solutions.