Vidéo de la conférence de Jean-Philippe Magué (ENS Lyon) & Jian-Yun Nie (U de Montréal)

Value of the Information & Strength of the Argument

The value of the information is high, as it synthesizes recent empirical findings and theoretical debates. The argumentation is solid: Magué systematically deconstructs the question, presents opposing views, and uses concrete examples to support his thesis that LLMs construct representations. He avoids overclaiming, acknowledging the complexity and open questions. The reasoning is clear and accessible, though it relies on the audience’s familiarity with concepts like modular addition and the distributional hypothesis.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor by referencing specific studies (Nanda et al., Karvonen, Anthropic) and theoretical frameworks (distributional hypothesis, symbol grounding). The sources are credible and relevant. The title accurately reflects the content, which is a focused exploration of whether LLMs can signify. The presentation is well-structured, with clear transitions and a balanced treatment of different perspectives.

The title accurately reflects the content, which focuses on whether LLMs can signify or understand meaning.

8/10

The talk is given by a researcher in computational linguistics, with references to empirical studies (Nanda et al., Karvonen, Anthropic) and theoretical frameworks (distributional hypothesis, symbol grounding). The arguments are nuanced and well-structured, though the presentation is an opinion/expert talk rather than a peer-reviewed study.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction by Michel Sinatra, director of CRIHN, presenting the seminar series and the speaker.
• 03:00Jean-Philippe Magué introduces the topic: do LLMs understand what they read and say? He deconstructs the question's terms.
• 10:00Discussion of the divide in the research community on whether LLMs understand, referencing a 2022 survey.
• 20:00Magué argues against the 'statistical regularities' objection, using the example of Babylonian eclipse prediction.
• 30:00Presentation of Nanda et al.'s study on modular addition, showing that the model builds angular representations.
• 40:00Karvonen's chess experiment: a language model trained on PGN notation develops an internal board representation.
• 50:00Anthropic's research on Claude: the model tracks line breaks by representing the overall structure of text.
• 1:00:00Discussion of the distributional hypothesis and its relevance to LLMs.
• 1:10:00Introduction of the symbol grounding problem and its implications for LLM understanding.
• 1:20:00Conclusion and transition to the next session's topic.

• International AI Safety Report — Magué cites this report for benchmarks showing LLM performance improvements.
• Nanda et al. (2023) - Progress measures for grokking on modular addition — Study demonstrating that a model learns modular addition by constructing angular representations.
• Karvonen - Chess game representation in language models — Work showing that a language model trained on chess PGN notation builds an internal board representation.
• Anthropic - Research on Claude's line break prediction — Study revealing that Claude tracks line breaks by representing the overall structure of text.

• Nanda et al. (2023) - Progress measures for grokking on modular addition — Supports the claim that LLMs build internal representations.
• Karvonen - Chess game representation in language models — Further evidence of internal world models in LLMs.
• Anthropic - Research on Claude's line break prediction — Demonstrates that LLMs track structural information, supporting the idea of meaning construction.

• Bender & Koller (2020) - Climbing towards NLU: On Meaning, Form, and Understanding — This paper argues that form alone is insufficient for meaning, challenging the distributional hypothesis as applied to LLMs.