CS 471/671 (How to grow language models) - #20 - Fall 2026

CS 471/671 (How to grow language models) - #20 - Fall 2026

Title history
  • CS 471/671 (How to grow language models) - #20 - Fall 2026 — observed on 20261113-0838 UTC (YouTube)
🎙 Center for Language & Speech Processing (CLSP), JHU 👥 4K 📅 November 13, 2026 ⏱ 39 min 👁 17 📄 tutorial 🧭 2026-11-13
Available in: English (current) Français

Keywords

policy gradientRLHFGRPOKL divergenceimportance samplingoff-policyreward functionadvantage

Summary

This lecture from a graduate course on self-supervised learning provides a comprehensive recap of policy gradient methods in the context of reinforcement learning from human feedback (RLHF). The instructor defines Kullback-Leibler divergence formally and uses visual examples to illustrate its role in constraining the policy model. The discussion then shifts to the evolution from RLHF to modern approaches like GRPO, emphasizing verifiable rewards and reasoning-oriented training. Key modifications include the use of advantage functions to reduce variance and the adoption of off-policy updating to handle expensive long rollouts. The lecture demonstrates how to correct distributional mismatch through importance weighting. A concrete example from DeepSeek’s reward function is analyzed. The lecture concludes with a corrected algorithm that incorporates a policy ratio to enable multiple gradient updates per rollout. Throughout, interactive pulse checks reinforce understanding of reward functions, baselines, and policy optimization. The content bridges foundational theory with recent advances in large language model training, making it highly relevant for students and practitioners.

161 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture delivers substantial value by systematically unpacking the mathematics behind policy gradients and their application in contemporary LLM training. The argumentation is logically sequenced: starting with the KL divergence as a regularization term, the instructor explains its properties and intuition before moving to concrete examples. The transition from RLHF to GRPO is well-motivated, highlighting practical limitations such as the cost of long rollouts and the shift toward verifiable rewards. The introduction of advantages and importance sampling is mathematically justified, and the final algorithm is presented as a coherent improvement. However, the lecture remains at a tutorial level, reproducing known ideas rather than presenting new research. The reasoning is solid and supported with clean derivations, though some claims about specific methods (e.g., DeepSeek’s reward functions) are stated without explicit citation, and the audience is expected to have prior familiarity with RL concepts.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the lecturer is from JHU’s CLSP, and the course is part of a formal graduate program. The content aligns well with current literature on RLHF and GRPO, though no external references are explicitly cited during the lecture. The only source provided is the course website, which contains syllabus and supplementary materials. The title “How to grow language models” aptly captures the process of training and refining LLMs, but it is somewhat broad; the lecture focuses specifically on reward-based optimization rather than all growth aspects. The inclusion of interactive questions ensures engagement and checks understanding, though a deeper exploration of certain topics (e.g., advantage estimation) is deferred. Overall, the lecture is scientifically trustworthy, but the lack of direct citations within the video limits its standalone rigor.

287 words

Title / Content Match

Title accurately reflects the lecture's focus on practical and theoretical methods for scaling language models, particularly via RLHF and GRPO.

Quality & Reliability

8/10

Academic lecture from a leading research institution, presenting formal definitions and derivations with clear pedagogical intent. Content aligns with established literature on RL and LLM training.

Key Moments

Cited Sources

  • Course Website — Official course page providing syllabus and materials

Concurring Sources

Contribution & Novelties

The lecture provides a clear pedagogical bridge between canonical RLHF and modern GRPO, emphasizing the practical motivation for each modification (e.g., verifiable rewards, off-policy learning, importance weighting). Its primary novelty lies in synthesizing recent algorithmic changes in a step-by-step manner, making them accessible to advanced students. The explicit treatment of long rollout cost and the accompanying algorithmic adjustments is particularly valuable.

Pour aller plus loin :

113 words

Radar Profile

The radar profile shows high scores for technical level and reliability, reflecting the academic rigor and depth of the content. The information quantity and quality are moderate to high, but the novelty is limited as it is a tutorial rather than a research presentation. This results in a slightly elongated shape with a strong bias toward expertise and trustworthiness.

Reliability 8/10