LLMs in Science: The Good, The Bad and The Ugly - Nihar Shah

Value of the Information & Strength of the Argument

The talk provides valuable insights into the application of LLMs in peer review, supported by empirical data from controlled experiments. Shah presents concrete examples and statistics, such as the low detection rate of errors by human reviewers (only 1 out of 79 identified a subtle error) and the bias in author evaluations. The argumentation is solid, with clear explanations of experimental designs and limitations. He also discusses the broader implications for scientific integrity, making the content highly relevant for researchers and policymakers.

Scientific Rigor, Source Quality, Title Accuracy

Shah demonstrates scientific rigor by referencing published studies and ongoing research, including experiments at NeurIPS 2022 and collaborations with program chairs. He acknowledges limitations, such as the generalizability of single-paper experiments. The title accurately reflects the content, and the talk is well-structured. The sources cited are credible, and the speaker’s expertise adds to the reliability. The adéquation between title and content is strong, with no significant discrepancies.

The title accurately reflects the three-part structure of the talk, covering positive, negative, and problematic aspects of LLMs in science.

8/10

The talk presents empirical studies and randomized controlled trials from peer-reviewed venues, with detailed methodology and transparent limitations. The speaker is a recognized expert in the field, and the content is grounded in published research, though some claims are based on unpublished or ongoing work.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction and overview of the talk's structure: The Good, The Bad, The Ugly.
• 05:00Discussion on evaluating AI reviewers and the challenge of matching human reviews.
• 10:00Experiment with 79 reviewers: only 1 identified a subtle error, highlighting human review limitations.
• 15:00Analysis of review quality and the lack of correlation with self-reported confidence.
• 20:00Experiment at NeurIPS 2022 showing authors' bias in evaluating reviews.
• 25:00Randomized controlled trial demonstrating that longer reviews receive higher scores, indicating style bias.
• 30:00Comparison of LLM and human error detection: GPT-4 found obvious errors but missed subtle ones without steering.
• 35:00Introduction to fraud in peer review: fake accounts and identity theft.
• 40:00Collusion rings and the vulnerability of reviewer assignment systems.
• 45:00Attack vectors on text-matching-based assignment: modifying abstracts, updating profiles, and using past data.

• OpenReview — Mentioned as a platform with publicly available reviews and as a venue where fake profiles were found.
• NeurIPS 2022 — The venue where the randomized controlled trial on review evaluation was conducted.
• ICLR — Mentioned as a conference with publicly available past data, used for training attacks.
• CVPR — Mentioned as a venue that uses only text matching for reviewer assignment.
• ARR (Association for Computational Linguistics Rolling Review) — Mentioned as a venue that has banned bidding and uses only text matching.

• The Science of Peer Review — An article discussing the challenges and potential improvements in peer review, aligning with the talk's themes.
• AI for peer review — An article exploring the use of AI in peer review, supporting the potential benefits and risks discussed.

• LLMs are not reliable reviewers