Do World-Model Agents Beat Model-Free LLM Agents on ARC-AGI-3-Style Interactive Tasks?

ARC-AGI-3 tests agents on novel turn-based games whose rules, goals and controls are never stated, so an agent has to explore, work out how the environment behaves, and then act efficiently. This problem compares two agent families under one matched budget: agents that learn an explicit model of the game's dynamics and plan with it, and LLM agents that choose actions directly from observations with no explicit model. The question is which family completes more levels with fewer actions, and where each one fails.

Originator:theo-panaComments: 0Created: Oct 21, 2026, 05:00 PMLast edited: Oct 21, 2026, 05:00 PM

0

Evaluation plan

Run each agent (model-free LLM agent, world-model agent with planning, random baseline, and optionally the hybrid ablation) on every game with the same cap on environment actions and the same compute or token budget. Use at least 3 seeds per agent per game, and keep a held-out subset of games unseen during development. Primary metrics: levels completed per game, and environment actions per completed level. Secondary: compute or token cost per completed level, next-frame prediction accuracy of the learned world model, and a failure analysis grouped by the mechanic that caused failure. Report a per-game results table with mean and spread, curves of levels completed against actions taken, and results at two or three budget levels. The question is answered in favour of world models only if they win on held-out games at matched budget.

Datasets / Resources

Public ARC-AGI-3 games via the official ARC Prize API and agents toolkit (arcprize.org)
Open re-implementations of ARC-AGI-3-style grid games, if API access is limited

Discussion

Sign in to comment

No comments yet.

Machine Learning / Calibration and RobustnessAdvancedBenchmark / Challenge

Does Basic Income Increase Productivity and GDP per Capita in the Issuing Country?Economics / Public PolicyAdvancedOpen Question

Testing Learning--Reasoning Correspondence as a Sample-Efficient Inductive Bias for Small Reasoning AgentsExploratoryOpen Question

Counterfactual Feature Interventions for Robust Tabular Reasoning in Small Language ModelsExploratoryOpen Question

Do World-Model Agents Beat Model-Free LLM Agents on ARC-AGI-3-Style Interactive Tasks?

Workspace

Problem Statement

Workspace

Problem Statement

Evaluation plan

Datasets / Resources

Tags

Discussion