CS 471/671 (How to grow language models) - #21 - Fall 2026

CS 471/671 (How to grow language models) - #21 - Fall 2026

Title history
  • CS 471/671 (How to grow language models) - #21 - Fall 2026 — observed on 20261113-0838 UTC (YouTube)
🎙 Center for Language & Speech Processing (CLSP), JHU 👥 4K 📅 November 13, 2026 ⏱ 49 min 👁 11 📄 tutorial 🧭 2026-11-13
Available in: English (current) Français

Keywords

RLHFGRPOKL divergencepolicy optimizationreward model

Summary

The lecture provides a detailed walkthrough of reinforcement learning from human feedback (RLHF) and its evolution into the GRPO (Group Relative Policy Optimization) algorithm. The instructor first explains the need for a KL divergence regularization term in the original RLHF to prevent the policy from drifting too far from the reference model, and clarifies the distinction between the three policies: target, reference, and old. The discussion then covers importance weighting and the instability caused by reusing the same rollout batch, leading to the introduction of a clipped objective to control the policy ratio. The lecture then transitions to GRPO, which samples multiple responses per prompt, normalizes rewards into advantages within a group, and uses a clipped objective with a KL penalty. Practical considerations are highlighted, such as selecting an appropriate dataset that lies in the zone of proximal development, monitoring reward diversity within groups to ensure learning, and addressing a length bias that encourages overlong responses. The instructor also notes the difficulty bias from advantage normalization and mentions that recent work sometimes omits the denominator. The session concludes with a visual interpretation of the GRPO objective and a Q&A segment.

190 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture is highly valuable for its clarity in unpacking the mathematical and conceptual underpinnings of RLHF and GRPO. The instructor builds a logical narrative, starting from the basic RLHF objective, identifying its weaknesses (policy drift, importance weight instability), and then systematically introducing modifications that culminate in the GRPO algorithm. The argumentation is rigorous and well-justified, with concrete examples and visual aids that enhance understanding. The instructor also anticipates common questions and corrects misconceptions (e.g., the difference between the old and reference policies), making the presentation particularly effective for learners. The use of analogies, such as the zone of proximal development, further strengthens the argumentation by linking abstract concepts to familiar educational principles.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is part of a university course (CS 471/671) and is grounded in established research, particularly the DeepSeek paper that introduced GRPO. The instructor consistently references the notation and formulations from that paper, ensuring alignment with the literature. The course page linked in the description provides additional context and materials, enhancing the credibility of the content. The title ‘How to grow language models’ is somewhat ambiguous, but the lecture effectively addresses training and scaling techniques, though it focuses more on fine-tuning than on model growth in the sense of architecture or size. Overall, the rigor is high, and the sources, while not exhaustively cited in the transcript, are sufficient for the depth of explanation provided.

244 words

Title / Content Match

The title 'How to grow language models' is somewhat broad, but the lecture focuses on training and fine-tuning techniques for language models, which aligns with the title.

Quality & Reliability

8/10

Lecture provides clear, accurate explanations of RLHF and GRPO concepts, based on established research. Instructor addresses student questions and clarifies complex topics, making it a reliable educational resource.

Key Moments

Cited Sources

Concurring Sources

Contribution & Novelties

This lecture offers a clear and comprehensive pedagogical explanation of GRPO, highlighting practical pitfalls that are often glossed over in research papers, such as reward diversity, difficulty bias, and length bias. It bridges the gap between the formal algorithm and its implementation, providing actionable advice for practitioners. The use of visualizations and analogies makes the material accessible, while still maintaining technical depth.

Pour aller plus loin :

143 words

Radar Profile

The radar profile shows high quality and technical depth, with balanced scores reflecting strong information content and reliability. The relatively lower quantity score suggests the lecture is focused and not exhaustive, but the quality and technical rigor compensate, resulting in a solid educational resource.

Reliability 8/10