Do AI Systems Have World Models? Probing Reasoning, Forecasting, and Generalization

The talk provides a rigorous and thought-provoking framework for assessing world models in AI, grounded in cognitive science. Ginosar’s use of Craik’s definition is insightful, shifting the focus from internal representation to behavioral affordances. The distinction between reasoning and forecasting as two modes of simulation is elegant and clarifies how to test for world models. The spatial reasoning benchmark is well-designed, leveraging walking tour videos and automatic map grounding to create a realistic and challenging evaluation. The finding that VLMs excel at landmark recognition but fail at path integration is significant, suggesting they rely on visual similarity rather than building a coherent spatial representation. This is a valuable contribution to the ongoing debate about whether AI systems truly understand the world or merely mimic surface patterns. However, the talk has limitations. The second line of work on motion forecasting is only briefly mentioned, leaving the audience with an incomplete picture. The evaluation of model behavior relies on indirect inference, and the speaker acknowledges the challenge of interpreting what models are ‘doing’ internally. The talk does not address potential confounding factors, such as the models’ prior exposure to similar data. Despite these caveats, the talk is scientifically sound, well-structured, and offers a clear methodology for probing world models. The sources cited are appropriate, including the speaker’s own research and foundational cognitive science literature. The title accurately reflects the content. Overall, this is a high-quality talk that advances the discussion on AI world models.

The title accurately reflects the content: the talk probes whether AI systems have world models by examining reasoning, forecasting, and generalization.

8/10

The talk is given by an assistant professor at TTIC, based on peer-reviewed research (including work with Google DeepMind). It references established concepts (Craik's mental models, Tolman's cognitive maps) and presents a clear framework. However, it is a conference talk, not a peer-reviewed publication, and some claims (e.g., model behavior) are based on specific experiments that are not fully detailed in the talk.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction: AI systems only see human-created artifacts, not the real world.
• 02:00Craik's definition of mental models: reasoning about alternatives, forecasting, and generalization.
• 05:00Norman's critique of human mental models: incomplete, unstable, unscientific.
• 07:00Formalization: world model as state + dynamics, enabling simulation.
• 09:00Reasoning and forecasting as two ways of using the same model.
• 11:00Spatial reasoning benchmark: walking tour videos and three levels of spatial understanding.
• 15:00Results: VLMs excel at landmark recognition and loop closure, but fail at path integration.
• 20:00Discussion: models rely on visual search, not path integration.

• Simons Institute talk page — Official page for the talk, providing details and possibly slides.

• Simons Institute talk page — Official page for the talk, providing details and possibly slides.