Behavioral evaluation of language models as models of human sentence processing

Value of the Information & Strength of the Argument

The talk provides valuable insights into the intersection of LLMs and psycholinguistics, offering a clear theoretical framework (rational analysis) and empirical evidence. The argumentation is solid, grounded in published research and logical reasoning. Levy effectively demonstrates how surprisal from LLMs aligns with human reading times, supporting the validity of these models as cognitive models. He also critically examines limitations, such as the need for direct probability measures over prompting. The presentation is well-structured, moving from theoretical foundations to concrete examples and recent findings.

Scientific Rigor, Source Quality, Title Accuracy

The talk is scientifically rigorous, with references to peer-reviewed publications (e.g., Hu & Levy 2023, Shain et al. 2024, Wilcox et al. 2023, Futrell et al. 2020). The sources are credible and directly support the claims. The title accurately reflects the content, focusing on behavioral evaluation of language models. The presentation is well-organized and the speaker’s expertise is evident. No comments were provided, so no analysis of public reception is included.

The title accurately reflects the content, which focuses on behavioral evaluation methods for language models in the context of human sentence processing.

9/10

Presentation by a leading expert (MIT professor, president of Cognitive Science Society) with references to peer-reviewed publications. The talk is a synthesis of established and recent research, with clear methodological explanations.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction to the talk and the concept of expectation in language.
• 10:00Explanation of large language models and their capabilities in language processing.
• 20:00Discussion on the incrementality of language comprehension and the role of prediction.
• 30:00Introduction to the rational analysis framework and Bayesian inference in language processing.
• 40:00Explanation of surprisal theory and its relationship to processing difficulty.
• 50:00Demonstration of how LLMs can be used to test syntactic knowledge and predict reading times.
• 1:00:00Discussion of the limitations of LLMs and the importance of direct probability measurements.
• 1:10:00Conclusion and implications for cognitive science and natural language processing.

• Prompting is not a substitute for probability measurements in large language models — Cited as evidence that direct probability measurements are better than prompting for evaluating LLMs.
• Large-scale evidence for logarithmic effects of word predictability on reading time — Cited as evidence for the relationship between surprisal and reading times.
• Using Computational Models to Test Syntactic Learnability — Cited as work using computational models to test syntactic knowledge.
• Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing — Cited as a model of memory effects in sentence processing.

• Large-scale evidence for logarithmic effects of word predictability on reading time — Supports the claim that surprisal correlates with reading times.
• Prompting is not a substitute for probability measurements in large language models — Supports the claim that direct probability measurements are superior to prompting.