Stanford CS229 Machine Learning | Spring 2026 | Lecture 20: GMM (EM), PCA

This lecture provides a rigorous and detailed treatment of policy gradient methods, a cornerstone of modern reinforcement learning. The instructors, Chris Ré and Tengyu Ma, are renowned experts in the field, and their explanations are mathematically precise yet accessible to an audience with a solid foundation in probability and calculus. The derivation of the policy gradient estimator is thorough, and the simplification to the ‘reward-to-go’ form is clearly motivated, highlighting the intuition behind why past rewards should not influence the gradient. The introduction of PPO is timely, given its widespread use in training large language models, and the lecture effectively connects theoretical concepts to practical applications.

The coverage of GMM and EM is also well-executed, with clear derivations and explanations of the algorithm’s convergence properties. The inclusion of PCA provides a comprehensive overview of dimensionality reduction techniques, complementing the probabilistic models discussed earlier. The lecture’s structure is logical, progressing from foundational concepts to advanced applications, and the instructors make effective use of examples and visual aids.

One potential limitation is the lack of discussion on the limitations or potential pitfalls of these methods, such as the high variance of policy gradient estimators or the local optima issues in EM. Additionally, the lecture assumes a high level of mathematical maturity, which may be challenging for beginners. However, for its intended audience, the content is of high quality and aligns well with the course’s objectives.

The title accurately reflects the content, as the lecture indeed covers GMM, EM, and PCA, along with policy gradient and RL for LLMs. The video is part of a reputable academic series, and the description provides links to official course materials, enhancing its credibility. Overall, this lecture is a valuable resource for anyone seeking a deep understanding of these machine learning techniques.

The title accurately reflects the content, which covers GMM (EM) and PCA, as well as policy gradient and RL for LLMs.

8/10

Lecture from Stanford University's CS229 course, delivered by professors Chris Ré and Tengyu Ma, known for their expertise in machine learning. The content is mathematically rigorous, with derivations and explanations of policy gradient methods. The video is part of a reputable academic series, and the description provides links to official course materials. However, the video is a lecture, not peer-reviewed research, and the transcription may contain minor errors.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and agenda: policy gradient, PPO, and RL for LLMs.
• 02:00Review of policy gradient algorithm and derivation of gradient estimator.
• 10:00Simplification of policy gradient using the fact that expected gradient of log policy is zero.
• 20:00Derivation of reward-to-go and removal of past rewards from gradient.
• 30:00Introduction of PPO algorithm and its importance in LLM training.
• 40:00Transition to GMM and EM algorithm, with mathematical formulation.
• 50:00Detailed derivation of EM algorithm for GMM.
• 1:00:00Introduction to PCA and its connection to dimensionality reduction.
• 1:10:00Discussion on using RL to train LLMs for long reasoning tasks.
• 1:15:00Conclusion and wrap-up of the lecture.

• CS229 Course Website — Official course page with syllabus and materials.
• Stanford AI Professional Programs — Information about Stanford's AI programs.

• CS229 Course Website — Official course materials align with lecture content.