[M2L 2025] 3.2 Reinforcement Learning for LLMs - Jessica Hamrick

Value of the Information & Strength of the Argument

The talk provides a valuable synthesis of RL techniques for LLMs, clearly explaining the conceptual framework and differentiating between reward types. The argumentation is solid, building from foundational concepts to recent advances, and is supported by concrete examples (e.g., o1’s reasoning trace, DeepSeek-R1’s aha moment). The speaker effectively communicates the intuition behind PPO and DPO without oversimplifying, and she highlights practical considerations such as reward hacking and the complexity of online RL. The discussion of open questions adds depth and encourages further exploration.

Scientific Rigor, Source Quality, Title Accuracy

The talk demonstrates scientific rigor by referencing key papers (InstructGPT, DeepSeek-R1, DPO, chain-of-thought) and accurately describing their contributions. The sources are credible and appropriately cited. The title accurately reflects the content, which is a focused overview of RL for LLMs. The presentation is well-organized and technically accurate, though it does not delve into mathematical details, which is appropriate for a summer school lecture.

The title accurately reflects the content, which is a comprehensive overview of reinforcement learning techniques applied to large language models.

8/10

The talk is given by a researcher with extensive experience at DeepMind, providing a clear and accurate overview of RL techniques for LLMs. It covers established methods (RLHF, PPO, DPO) and recent developments (RLVR, RLEF, RLIF) with appropriate references to key papers. The presentation is well-structured and technically sound, though it remains at an introductory level without deep mathematical derivations.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and overview of the talk's structure.
• 02:00Discussion of OpenAI's o1 model and its reasoning capabilities.
• 05:00Introduction to the alignment problem and the need for RL.
• 10:00Explanation of supervised fine-tuning (SFT) and its role.
• 15:00Formulation of LLMs as RL agents in an MDP framework.
• 20:00Comparison of traditional deep RL and RL with LLMs.
• 25:00Overview of RLHF: preference data, reward model, and PPO.
• 30:00Explanation of PPO and its advantages over vanilla policy gradient.
• 35:00Introduction to DPO as an alternative to RLHF.
• 40:00Discussion of RLVR and its application to reasoning tasks.
• 45:00Exploration of RLEF for agentic tasks and tool use.
• 50:00Open research questions and concluding remarks.

• InstructGPT: Training language models to follow instructions with human feedback — Referenced for the alignment problem and RLHF.
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — Referenced for RLVR and reasoning models.
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Referenced for DPO.
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Referenced for chain-of-thought prompting.
• FLAN: Finetuning Language Models is a Compute-Optimal Way to Learn — Referenced for instruction tuning.

• OpenAI o1 System Card — Provides details on o1's reasoning and safety evaluations.
• DeepSeek-R1 Technical Report — Details the RLVR approach and results.