CAOS 2025 - 6 | Rovereto, May 7-9 | Jeffrey S. Bowers

Value of the Information & Strength of the Argument

The talk provides a valuable critical perspective on the use of neural networks in cognitive neuroscience. Bowers effectively argues that correlational approaches, such as brain score, are insufficient for establishing mechanistic similarity. He supports his claims with concrete examples, including his own research showing that brain score predictions are largely driven by background confounds, and he references studies where models fail on out-of-distribution stimuli that humans find easy. The argumentation is coherent and well-structured, though it is primarily an opinion piece rather than a systematic review. The speaker’s expertise in the field adds credibility, and he acknowledges the importance of experiments in psychology. However, the talk could benefit from a more balanced discussion of successful model-brain alignments, as it focuses heavily on failures.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor in its use of published studies and the speaker’s own research. Bowers cites specific papers, such as the PNAS study on transformer models and the Vong et al. Science paper on language acquisition, and he discusses the limitations of these studies. The quality of sources is high, and the arguments are based on empirical evidence. The title accurately reflects the content, which is a critical examination of neural network models. The talk does not include a discussion of public comments, as none were provided.

The title accurately reflects the content, which focuses on deep problems with neural network models of human vision and language.

8/10

The talk presents a critical perspective on neural network models of vision and language, supported by references to published studies and the speaker's own research. The arguments are logically structured and based on empirical evidence, though the presentation is an opinion piece rather than a systematic review.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction: Bowers outlines his thesis that claims about neural networks are exaggerated.
• 05:00Discussion of standard scientific method: experiments manipulating independent variables.
• 10:00Critique of brain score benchmarks: correlational studies and lack of causal evidence.
• 15:00Example of confounds in brain score: background vs. object identity.
• 20:00Behavioral benchmarks and out-of-distribution failures.
• 25:00Critique of strong claims from weak findings in language models.
• 30:00Discussion of language acquisition claims and the role of innate biases.
• 35:00Conclusion: need for experiments and caution in interpreting correlations.

• CIMeC CAOS Workshop — Workshop page providing context for the talk.

• Brain-Score — Benchmark used in the talk to illustrate correlational approaches.
• Geirhos et al. (2018) on texture bias — Study showing CNNs rely on texture, supporting the critique of model mechanisms.

• Schrimpf et al. (2018) Brain-Score — The Brain-Score paper claims that models can predict brain responses, which Bowers argues is overstated due to confounds.