[M2L 2025] 3.3 Context: from context to capabilities - Max Bartolo

Value of the Information & Strength of the Argument

The talk provides valuable insights into the evolution of context handling in NLP, from early RNN-based systems to modern transformer architectures. The speaker’s personal experience adds authenticity, and the explanation of RAG and in-context learning is clear and well-argued. The argumentation is solid, supported by references to key papers and models, though some points are based on anecdotal evidence. The discussion of the trade-offs between parametric and non-parametric knowledge is particularly insightful.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor by referencing seminal works such as the RAG paper, GPT-3, and BERT, and by explaining the underlying mechanisms. The speaker also mentions his own research on influence functions and in-context learning, which adds credibility. The title accurately reflects the content, focusing on the role of context in LLMs. The presentation is well-structured and technically accurate, though it does not provide formal citations or a bibliography.

The title accurately reflects the content, which focuses on the evolution and role of context in large language models, from early retrieval systems to modern in-context learning.

8/10

The talk is given by a researcher at Google DeepMind with direct experience in the field, providing a historical and technical overview of context in LLMs. The content is well-structured, references key papers and models, and includes personal insights. However, it is a lecture rather than a peer-reviewed source, and some claims are based on personal experience.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction to retrieval-based question answering and early work at Bloomfire AI.
• 05:00Explanation of BiDAF architecture and its attention mechanisms.
• 10:00Discussion of RNN limitations: memory decay and recency bias.
• 15:00Introduction of transformer-based models (GPT, BERT) and their fixed context windows.
• 20:00Explanation of pre-training objectives and what models learn from next-word prediction.
• 25:00Introduction of retrieval-augmented generation (RAG) and its benefits.
• 30:00Discussion of modern RAG integration and flexibility.
• 35:00Use of long documents and conversational agents as context.
• 40:00Explanation of in-context learning and its sensitivity to example ordering.
• 45:00Conclusion: three stages of learning in LLMs and the role of context.

• BiDAF: Bidirectional Attention Flow for Machine Comprehension — Mentioned as the machine reading model used in the early demo.
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Mentioned as a key transformer-based model with a 512-token context window.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Introduced RAG and its approach to combining retrieval and generation.
• Language Models are Few-Shot Learners — Introduced GPT-3 and in-context learning.
• What Makes In-Context Learning Work? — The speaker's own work on in-context learning and example ordering.

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — The talk's description of RAG aligns with the paper's content.
• Language Models are Few-Shot Learners — The talk's explanation of in-context learning matches the paper's findings.