Rethinking Research: The Role of Humans in Scientific Discovery in the Age of LLMs

Value of the Information & Strength of the Argument

The talk provides valuable insights into the practical integration of LLMs in research and supervision, grounded in the speaker’s extensive experience. The argumentation is coherent and well-structured, moving from personal observations to broader implications. The speaker effectively uses examples to illustrate his points, such as the AI Co-Scientist’s ability to generate novel hypotheses. He also addresses potential concerns, such as the risk of feeling inferior to LLMs, and offers a balanced perspective on automation. However, the argument is largely anecdotal and lacks systematic empirical evidence, which limits its generalizability.

Scientific Rigor, Source Quality, Title Accuracy

The speaker demonstrates scientific rigor by acknowledging the limitations of LLMs and emphasizing the need for human oversight. He references specific tools and studies, such as DeepMind’s AI Co-Scientist and a survey by Owkin, which adds credibility. However, he does not provide detailed citations or links to these sources, which would enhance verifiability. The title accurately reflects the content, and the talk stays on topic throughout. The speaker’s credentials and experience lend authority to his perspective.

The title accurately reflects the content, which focuses on the evolving role of humans in scientific discovery with the advent of LLMs.

8/10

The talk is given by a highly distinguished academic (FRS, FREng) with deep experience in research and supervision. It presents a balanced, thoughtful perspective on the role of LLMs in research, supported by personal examples and references to specific tools and studies. However, it is largely opinion-based and lacks systematic empirical evidence.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction by Shitij Kapur, setting the context of AI's impact.
• 05:00Speaker shares his background and initial use of ChatGPT for editing emails.
• 10:00Discussion of using LLMs to identify research problems and the need for domain expertise.
• 15:00Example of LLM generating research questions and the speaker's reaction of feeling inferior.
• 20:00Introduction of domain-specific LLMs and DeepMind's AI Co-Scientist.
• 25:00Discussion of autonomous laboratories and the trend towards automation.
• 30:00Survey by Owkin on industry attitudes towards automation in drug discovery.
• 35:00Emphasis on critical thinking as a key human skill in the age of LLMs.
• 40:00Proposal of a three-way partnership between PhD students, supervisors, and LLMs.
• 45:00Conclusion: LLMs as co-creators, not co-authors, and the need for human accountability.

• AI Co-Scientist (DeepMind) — Mentioned as a tool for research collaboration that generates hypotheses.
• Owkin survey on automation in drug discovery — Referenced to show industry reluctance to trust automated science.

• AI Co-Scientist (DeepMind) — The speaker's positive view aligns with DeepMind's stated goal of augmenting experts.
• Owkin survey — The survey's findings support the speaker's argument that full automation is not yet trusted.

• Daniel Kokotajlo's prediction of superintelligence by 2027 — Contrasts with the speaker's more cautious view on the pace of AI adoption.
• Professor Narayanan's prediction of 40-year diffusion — Represents a slower timeline, but not necessarily discordant with the speaker's emphasis on human role.