RuCCS - Dr. Sean Trott - In Person Talk - Tuesday, February 10, 2026

Value of the Information & Strength of the Argument

The talk provides valuable insights into the potential and limitations of using LLMs to study human cognition. The argumentation is solid: the speaker carefully designs experiments, controls for confounds, and compares model behavior to human behavior. He acknowledges the limitations of his approach and discusses alternative explanations. The use of log odds as a metric is well-explained, and the statistical modeling approach is appropriate. The talk also raises important epistemological questions about the validity of using LLMs as model organisms, which adds depth to the discussion.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor through careful experimental design, pre-registered attention checks, and transparent reporting of results. The speaker cites relevant literature on theory of mind and language models, and he acknowledges the limitations of his own work. The title accurately reflects the content, and the talk is well-structured. The speaker also discusses the importance of reproducibility and the need for open-weight models, which is a strength.

The title accurately reflects the content: a talk by Dr. Sean Trott at RuCCS on using language models to study false belief reasoning.

8/10

The talk presents original research with clear methodology, including controlled stimuli, human comparison, and statistical analyses. The speaker acknowledges limitations and discusses epistemological challenges. However, the talk is a presentation of ongoing work, not a peer-reviewed publication, and some details are simplified.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 05:00Introduction to the talk and the research program 'LLMology'.
• 10:00Discussion of competing hypotheses for the origin of theory of mind.
• 15:00Explanation of large language models and their training.
• 20:00Introduction to the false belief task and its adaptation for language models.
• 25:00Description of the custom false belief task design and stimuli.
• 30:00Presentation of GPT-3 results showing sensitivity to false beliefs.
• 35:00Human comparison study results and comparison with GPT-3.
• 40:00Discussion of limitations of closed-source models and the need for open-weight models.
• 45:00Replication with 41 open-weight LMs and findings on model size.
• 50:00Epistemological challenges in using LLMs as model organisms.

• Trott, S., et al. (2023). Do Large Language Models Know What Humans Know? — The speaker references his own published work from 2023, which is the basis of the first study presented.

• Trott, S., et al. (2023). Do Large Language Models Know What Humans Know? — The speaker's own published work, which is the primary source for the first study.