Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 9: Stochastic Dyn. Program

The lecture provides a rigorous and comprehensive introduction to stochastic dynamic programming, a cornerstone of optimal control and reinforcement learning. Prof. Pavone’s expertise is evident in the clear explanations and the careful handling of mathematical details. The content is well-structured, starting with the problem formulation, then deriving the Bellman equation, and illustrating with examples. The inventory control problem is particularly effective in demonstrating the mechanics of dynamic programming, including the handling of constraints and expectations. The stochastic LQR example elegantly shows that the certainty equivalence principle holds, with the optimal policy unchanged and only the cost affected by noise. The discussion of infinite-horizon MDPs and the Q-function is insightful, highlighting the practical importance of the Q-function in model-free settings. The lecture is technically rigorous, with references to Bertsekas’s textbooks for proofs, and the slides are available online. The main limitation is that it is a lecture, not a peer-reviewed source, but the quality is high. The title accurately reflects the content, and the lecture is suitable for advanced students or practitioners with a background in control theory. Overall, this is an excellent educational resource.

The title accurately reflects the content, which focuses on stochastic dynamic programming, including value iteration and policy iteration.

9/10

Lecture by a renowned expert in autonomous systems, with rigorous mathematical derivations and references to established textbooks. The content is well-structured and pedagogically sound, though it is a lecture rather than peer-reviewed research.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to stochastic dynamic programming and MDP formulation
• 10:00Explanation of Markov property and disturbance modeling
• 20:00Derivation of the Bellman equation for stochastic DP
• 30:00Inventory control example: problem setup and dynamic programming recursion
• 40:00Solving the inventory example by hand for state 0
• 50:00Stochastic LQR example: derivation and solution
• 1:00:00Introduction to infinite-horizon MDPs and discount factor
• 1:10:00Stationary Bellman equation and Q-function introduction

• AA203 Course Page — Course information and enrollment details.
• Principles of Robot Autonomy — Companion textbook for the course.
• Course Schedule and Syllabus — Schedule and syllabus for the course.
• Lecture Slides — Slides for this lecture.
• Full Playlist — Playlist of all lectures.

• Dynamic Programming and Optimal Control by Dimitri Bertsekas — Standard reference for dynamic programming, cited in the lecture.