Reinforcement Learning 2026 - Session 15

Reinforcement Learning 2026 - Session 15

by Robust and Interpretable Machine Learning Lab

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQMMachine learning
🎙 Robust and Interpretable Machine Learning Lab 👥 1K 📅 July 13, 2026 ⏱ 84 min 👁 6 📄 lecture 🧭 2026-08-16
Available in: English (current) Français

Keywords

UCBregretHoeffdingcontextual banditupper confidence bound

Summary

This session of a reinforcement learning course begins by clarifying the previous discussion on contextual bandits, emphasizing that the context vector contains only user features, while action-specific parameters are learned separately. The main focus is on deriving a theoretical upper bound for the regret of the UCB algorithm. The instructor defines key notation: Q(a) as the expected reward for action a, Q_hat(a) as the empirical estimate, and U(a) as the upper confidence bound. The regret is expressed as the sum over time of the difference between the optimal action’s value and the chosen action’s value. By adding and subtracting the UCB term, the regret is decomposed into two parts. Under a condition based on the Hoeffding inequality, the first part is shown to be negative, allowing it to be dropped. The condition requires that the empirical mean is within a certain bound of the true mean for all actions and times, which is ensured with high probability by scaling the confidence parameter as delta/t^2. This leads to a union bound argument, resulting in a probability of at least 1 - O(delta). The second part of the regret is then bounded by substituting the definition of U, yielding a deterministic term plus a stochastic term involving the difference between the empirical and true means. This difference is again bounded using the same condition, leading to a final regret bound of O(sqrt(K T log T)). The lecture is theoretical and includes interactive Q&A, with some audio interruptions.

245 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture provides a rigorous and detailed derivation of the UCB regret bound, which is a fundamental result in bandit theory. The argumentation is logically structured, starting from the definition of regret and progressively building up to the final bound. The use of the Hoeffding inequality and the union bound is standard and well-explained. The instructor also addresses potential pitfalls, such as the difference between pointwise and uniform confidence bounds, and justifies the scaling of the confidence parameter. The value of the information is high for students or researchers seeking a deep theoretical understanding of bandit algorithms.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is scientifically rigorous, with clear definitions and proofs. However, no external sources are cited in the video or description, so the quality of sources cannot be assessed. The title accurately reflects the content, which is a session on reinforcement learning focusing on bandits. The content is consistent with established literature on multi-armed bandits, though the lack of citations is a minor weakness.

176 words

Title / Content Match

The title accurately reflects the content: a session on reinforcement learning, specifically focusing on contextual bandits and the theoretical analysis of UCB regret.

Quality & Reliability

7/10

The lecture provides a rigorous theoretical derivation of the regret bound for UCB, with clear notation and step-by-step proofs. The content is consistent with standard bandit theory, though the video is a recording of a live session with some audio interruptions.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Contribution & Novelties

The lecture provides a clear and detailed derivation of the UCB regret bound, which is a classic result. The novelty lies in the pedagogical approach, breaking down the proof into manageable steps and addressing common misconceptions. It also highlights the importance of scaling the confidence parameter to achieve a high-probability bound.

Pour aller plus loin :

87 words

Radar Profile

The radar profile shows high scores in technical level and quantity of information, reflecting the theoretical depth and comprehensive coverage of the topic. The quality of information and reliability are also strong, but slightly lower due to the lack of external citations and minor presentation issues.

Reliability 7/10