Venturing out into the world with AI

The talk provides a thoughtful and rigorous exploration of a central question in AI: whether LLMs possess world models. Kleinberg’s approach is commendable for grounding the discussion in formal methods, specifically the Myhill-Nerode theorem, which offers a principled way to define and extract states from sequence-generating machines. This is a significant contribution to the interpretability literature, as it moves beyond ad hoc probing to a theoretically motivated framework. The examples are well-chosen and illustrate the concepts clearly. The finding that a transformer trained on Manhattan taxi routes does not yield a coherent map when probed is striking and suggests that LLMs may rely on superficial patterns rather than deep structural understanding. However, the talk is limited by its scope: it focuses on discrete, finite-state domains, and the extension to more complex, continuous real-world scenarios is not addressed. Additionally, the negative result for Manhattan navigation is based on a specific training setup, and it is unclear how generalizable it is. The talk does not engage with alternative approaches to world model extraction, such as causal interventions or representation analysis. The discussion of the Myhill-Nerode theorem is accessible but assumes some familiarity with automata theory. Overall, the talk is intellectually stimulating and offers a fresh perspective, but it leaves many open questions and would benefit from a more comprehensive treatment of the challenges involved. The title is somewhat broad, but the content is well-aligned with the theme of AI venturing into the world. The presence of a short audience question adds value, but the talk would have been stronger with more interactive discussion.

The title is broad but accurately reflects the talk's focus on AI systems interacting with the world and the challenge of understanding their internal models.

8/10

Talk by a leading computer scientist at a prestigious institute, presenting research with clear methodology and references to prior work. However, it is a presentation, not a peer-reviewed publication, and some claims are anecdotal.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and tribute to Avrim Blum.
• 03:00Anecdote about navigating to Lipari and the need for an AI agent with a world model.
• 07:00Definitional question: what is a world model? Contrast between classical chess AI and LLMs.
• 10:00Introduction of the number-choosing game and its equivalence to tic-tac-toe via magic square.
• 13:00Explanation of the Myhill-Nerode theorem and its application to extracting states from sequences.
• 18:00Application to Manhattan taxi data: training a transformer to give directions and probing for a map.
• 25:00Discussion of results: the transformer does not have a coherent map of Manhattan.
• 30:00Implications for AI agents and the need for better understanding of LLM internal representations.
• 35:00Concluding remarks and future directions.

• Simons Institute talk page — Official page for the talk, providing context and possibly slides.

• Myhill–Nerode theorem — The theorem is used as the basis for the state extraction method.