Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

The lecture provides a comprehensive and rigorous overview of train-time scaling techniques, grounded in three influential papers. The instructor, Aakanksha Chowdhery, is a recognized expert with hands-on experience in training large models, lending credibility to the presentation. The content is well-structured, starting with motivation via the AIME benchmark, then detailing each paper’s methodology and contributions, and finally discussing broader implications and open questions. The technical depth is high, with clear explanations of concepts like GRPO, entropy collapse, and dynamic sampling, making it suitable for an audience with some background in machine learning. The lecture also includes a valuable discussion on the trade-offs between train-time and test-time compute, and the importance of verifiability in domains like math and code. However, the lecture is a single perspective and does not critically evaluate the limitations of the presented methods beyond what is mentioned in the papers. The instructor also acknowledges that some plots may be misleading, which adds a layer of nuance. Overall, the lecture is a valuable resource for understanding state-of-the-art techniques in self-improving AI agents, with a strong emphasis on practical implementation details. The adéquation between title and content is excellent, as the lecture directly addresses train-time scaling and scaling RL. The sources cited are the papers themselves, which are peer-reviewed and well-regarded in the field. The lecture does not include any advertising or sponsored content. The audience appears to be graduate students or researchers, but the content is presented clearly enough for a broader technical audience. The discussion of open questions encourages further exploration and critical thinking. Overall, this is a high-quality lecture that provides both theoretical insights and practical guidance.

The title accurately reflects the content, which focuses on train-time scaling and scaling reinforcement learning.

8/10

Lecture from Stanford University by an expert in the field, covering peer-reviewed papers (STaR, DeepSeekMath, DAPO) with technical depth. The content is well-structured and based on established research, though it is a single perspective and not peer-reviewed itself.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to train-time scaling and overview of the three papers to be covered.
• 02:00Motivation using AIME benchmark, showing how smaller models can achieve high accuracy.
• 05:00Explanation of the three paradigms: pre-training, fine-tuning, and test-time scaling.
• 10:00Discussion on the trade-offs between train-time and test-time compute, referencing o1 results.
• 15:00Introduction to STaR paper: bootstrapping reasoning chains through rationalization and filtering.
• 30:00DeepSeekMath paper: GRPO and the importance of curated math data.
• 45:00DAPO paper: addressing entropy collapse and training instability in long chain-of-thought RL.
• 1:00:00Open questions and concluding remarks on why majority-at-K improves but pass-at-K does not.

• CS329A Course Page — Course syllabus and schedule.
• Agentic AI Professional Education Program — Related professional education program.
• CS329A Online Course — Online course offering.
• Course Playlist — Full lecture playlist.

• STaR: Self-Taught Reasoner — Paper discussed in the lecture.
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Paper discussed in the lecture.
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Paper discussed in the lecture.