Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 3: Policy Gradients

Item: Stanford CS224R Deep Reinforcement Learning | Spring 2025 | Lecture 3: Policy Gradients
Rating: 5
Author: PostSingularity Institute

by Chelsea Finn

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQMMachine learning

🎙 Chelsea Finn 👥 1.2M 📅 December 8, 2025 ⏱ 62 min 👁 21K 📄 lecture 🧭 2026-08-06

Available in: English (current) Français

Keywords

policy gradientreinforcement learninggradient descentneural networktrajectory

Summary

This lecture introduces policy gradients, a fundamental online reinforcement learning algorithm. The instructor, Chelsea Finn, begins by recapping the RL problem formulation and the limitations of imitation learning. She then outlines the online RL loop: initialize policy, collect data, improve policy, repeat. The core of the lecture is the mathematical derivation of the policy gradient. Starting from the objective of maximizing expected reward, she uses the log-derivative trick to transform the gradient of the expectation into an expectation of the gradient of the log policy times the reward. She explains how the gradient of the log trajectory simplifies to the sum of gradients of the log policy at each timestep, as the dynamics and initial state distribution do not depend on the policy parameters. The resulting vanilla policy gradient formula is presented, and its practical implementation is discussed: sample trajectories, compute rewards, compute gradients of log probabilities, and update the policy via gradient ascent. The lecture emphasizes intuition, implementation details, and when to use policy gradients, setting the stage for more advanced algorithms in subsequent lectures.

176 words

Critical Evaluation

The lecture is exceptionally clear and well-structured, making complex mathematical concepts accessible. The derivation of the policy gradient is presented step-by-step, with careful attention to the log-derivative trick and the simplification of the gradient of the trajectory probability. The instructor’s expertise is evident, and the content is accurate and up-to-date. The use of a concrete example (2D navigation) helps ground the theory. The lecture is part of a formal Stanford course, ensuring high academic rigor. The sources cited are the course website and Stanford Online, which are authoritative. The title accurately reflects the content. The only minor limitation is that the lecture is introductory and does not cover advanced variants or practical tricks in depth, but this is appropriate for the course level. Overall, this is an excellent educational resource.

130 words

Title / Content Match

The title accurately reflects the content: a lecture on policy gradients in deep reinforcement learning.

Quality & Reliability

9/10

Lecture by a renowned expert (Chelsea Finn) from Stanford University, part of a formal course. The content is mathematically rigorous, with clear derivations and practical insights. The source is highly credible (Stanford Online).

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

CS224R Course Website — Course syllabus and materials
Stanford Online Course Page — Enrollment information
Full Playlist — All lectures in the series

Concurring Sources

Policy Gradient Methods — General overview consistent with the lecture content
Reinforcement Learning: An Introduction — Standard textbook covering policy gradients

Contribution & Novelties

This lecture provides a clear and rigorous introduction to policy gradients, a cornerstone of modern reinforcement learning. It bridges the gap between theory and practice, offering both mathematical derivations and intuitive explanations. The lecture is particularly valuable for its step-by-step derivation of the policy gradient, which is often glossed over in other resources.

Pour aller plus loin :

Policy Gradient Methods — Overview of policy gradient methods and their variants.
Reinforcement Learning: An Introduction — The standard textbook by Sutton and Barto, covering policy gradients in depth.
Trust Region Policy Optimization — A popular advanced policy gradient algorithm.

98 words

Radar Profile

The radar profile shows high scores across all dimensions, indicating a well-balanced and comprehensive lecture. The strong scores in quantity and quality of information reflect the depth and clarity of the content, while the high technical level and reliability underscore its academic rigor.

Reliability 9/10