Stanford CS229 Machine Learning | Spring 2026 | Lecture 18: GMM (EM), PCA

This lecture provides a solid introduction to reinforcement learning, focusing on the foundational concepts of sequential decision-making and the Markov Decision Process (MDP). The instructor, Chris Ré, is a renowned professor, and the content is delivered with clarity and pedagogical effectiveness. The explanation of MDPs is thorough, covering states, actions, and transition dynamics, and the use of a simple 1D robot navigation example helps ground abstract concepts. The lecture then transitions to the policy gradient algorithm, a core method in RL, and derives the update rule in a clear, step-by-step manner. The mathematical presentation is rigorous, and the instructor takes care to explain the intuition behind the gradient ascent update. The lecture also touches on the exploration-exploitation trade-off, though it correctly notes that this is often not explicitly addressed in modern applications. One notable strength is the emphasis on the sequential nature of decision-making and the importance of long-term ramifications, which is a key differentiator from supervised learning. The lecture is well-structured and builds on previous knowledge, making it suitable for advanced students. However, there are a few limitations. The title of the video mentions GMM (EM) and PCA, but the content is actually about reinforcement learning, which is a significant mismatch that could confuse viewers. Additionally, the lecture does not provide external sources or citations, which is typical for a course lecture but limits its value as a standalone reference. The instructor also mentions that exploration-exploitation is not discussed in depth, which is a missed opportunity given its importance in RL. Overall, this is a high-quality lecture that effectively covers the basics of RL and policy gradient, but the title mismatch and lack of external references prevent it from being perfect. The lecture is likely to be very useful for students who are already familiar with machine learning concepts and are looking to expand their knowledge into RL.

The title mentions GMM (EM) and PCA, but the lecture actually covers reinforcement learning basics and policy gradient. This mismatch is significant.

8/10

Lecture from Stanford CS229, taught by renowned professors. Content is rigorous, well-structured, and based on established RL theory. However, it is a lecture, not peer-reviewed, and lacks external citations.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to reinforcement learning and sequential decision-making
• 03:00Discussion on exploration vs exploitation trade-off
• 07:00Introduction to Markov Decision Processes (MDP)
• 15:00Definition of states, actions, and transition dynamics
• 25:00Explanation of policy gradient algorithm
• 35:00Derivation of policy gradient update rule
• 45:00Discussion on reward functions and learning from rewards
• 55:00Example of policy gradient in a simple robot navigation task

• CS229 Course Website — Official course page for CS229, providing syllabus and materials.
• Stanford AI Programs — Information about Stanford's AI professional and graduate programs.

• Reinforcement Learning: An Introduction — Standard reference for RL, covering MDPs and policy gradient methods.
• Policy Gradient Methods for Reinforcement Learning with Function Approximation — Seminal paper on policy gradient methods.