CS 471/671 (How to grow language models) - #19 - Fall 2026

CS 471/671 (How to grow language models) - #19 - Fall 2026

Title history
  • CS 471/671 (How to grow language models) - #19 - Fall 2026 — observed on 20261113-0017 UTC (YouTube)
🎙 Center for Language & Speech Processing, JHU 👥 4K 📅 November 13, 2026 ⏱ 17 min 👁 2 📄 tutorial 🧭 2026-11-13
Available in: English (current) Français

Keywords

policy gradientreward hackingRLHFlanguage model trainingLLM fine-tuning

Summary

This lecture from JHU’s CS 471/671 course dives into the post-training phase of large language models, focusing on reinforcement learning (RL) after initial pre-training and instruction tuning. The instructor recaps that pre-training gives a next-token predictor, and instruction tuning aligns it with human-annotated examples. The core of the lecture is the policy gradient method, a fundamental RL technique used to optimize a policy model based on a learned reward function. The instructor explains the mathematical derivation of the gradient estimator, emphasizing the log-derivative trick and the impossibility of backpropagating through discrete sampling. A key concern is reward hacking, where the model exploits loopholes in an imperfect reward function, causing it to drift from the original policy. To mitigate this, a KL-divergence penalty term is added to the reward, discouraging large deviations from a reference policy. The lecture alternates between formal derivations and intuitive explanations, with a student interaction about learning rates and human annotation efficiency. Overall, it provides a solid pedagogical foundation for understanding RLHF.

165 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video delivers high pedagogical value by breaking down a complex RL algorithm into intuitive pieces. The instructor justifies each step logically, from the objective function to the gradient estimator, and gives clear intuition for why the policy gradient formula makes sense (weighting gradients by reward). The argumentation is solid, with formal derivations presented as bonus content, ensuring that even students who skip the math grasp the underlying logic. The discussion of reward hacking and the KL penalty is particularly valuable, as it addresses practical pitfalls in training LLMs and connects to broader RLHF practices. The lecture balances rigor with accessibility, making it a strong learning resource.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the content aligns with standard RL literature and is delivered by an academic institution. However, the lecture does not cite specific papers or external sources, aside from the course URL in the description. The title ‘How to grow language models’ is fitting, as the lecture addresses a critical stage in model development. No comments are provided, so no public feedback is analyzed. The lecture is self-contained and accurate, though it relies on prior knowledge from earlier lectures in the series.

207 words

Title / Content Match

Title adequately reflects content: the lecture covers a key step in growing (training) language models via reinforcement learning and policy gradient.

Quality & Reliability

8/10

Lecture from JHU CLSP, academically rigorous, clear explanations of policy gradient and reward hacking, though no citations to external sources presented.

Key Moments

Cited Sources

Contribution & Novelties

This lecture contributes a clear, step-by-step pedagogical exposition of policy gradient for RLHF, emphasizing the intuition behind the mathematical formulation and the practical problem of reward hacking. It stands out for explicitly connecting the gradient estimator to the KL regularization term and for addressing efficiency concerns in human annotation. While not presenting new research, it effectively synthesizes existing knowledge for learners.

Pour aller plus loin :

  • Reinforcement learning — Foundational concept underlying the entire lecture.
  • Policy gradient methods — Directly covers the algorithm taught, including variants like REINFORCE.
  • Reward hacking — The specific problem the KL penalty addresses.
  • Proximal Policy Optimization (PPO) — A widely used policy gradient variant in practice for LLM alignment.

114 words

Radar Profile

The radar profile shows strong scores in information quality and technical level, with slightly lower quantity and reliability, reflecting a focused lecture that prioritizes depth of explanation over breadth of topics, and relies on the instructor's expertise rather than external citations.

Reliability 8/10