What Makes a Good Proxy Reward Function for Language Model Post-Training?

Value of the Information & Strength of the Argument

The talk provides significant value by offering a novel optimization perspective on proxy reward functions, moving beyond the common accuracy metric. The argumentation is rigorous, with theoretical proofs for the role of reward variance and the potential benignity of certain reward errors. The speaker clearly explains the intuition behind the results and supports them with empirical evidence on large language models. The discussion of practical applications, such as the importance of supervised fine-tuning and data selection, adds further value. The argumentation is well-structured and persuasive, challenging conventional wisdom in a constructive manner.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high, with formal proofs and careful experimental validation. The speaker cites his own papers and mentions related work, though specific references are not detailed in the talk. The title accurately reflects the content, focusing on the properties of proxy reward functions. The talk is well-organized and the technical details are presented clearly. The audience questions are addressed thoughtfully, indicating a deep understanding of the subject. The overall quality of sources is strong, given the speaker’s expertise and the peer-reviewed nature of the underlying research.

The title accurately reflects the content, which focuses on the properties of proxy reward functions for post-training language models.

9/10

The talk presents rigorous theoretical results with proofs, supported by empirical demonstrations on large language models. The speaker is a recognized researcher with a strong publication record. The content is well-structured and addresses a fundamental question in RLHF.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction to language model post-training and the role of reinforcement learning.
• 05:00Definition of proxy reward functions and the problem of reward hacking.
• 10:00Presentation of the theoretical setting: policy parameterization and gradient flow.
• 15:00Introduction of reward variance and its connection to optimization speed.
• 20:00Proof that low reward variance leads to slow optimization, with illustration.
• 25:00Discussion of how accuracy does not capture the flatness of the objective.
• 30:00Empirical results on a 3B parameter model showing the impact of reward variance.
• 35:00Practical implications: supervised fine-tuning increases reward variance.
• 40:00Discussion of data selection and reward transformations based on the theory.
• 45:00Conclusion and summary of key takeaways.

• The Role of Reward Variance in Policy Gradient Optimization — Paper by Razin et al. (2024) proving the connection between reward variance and optimization speed.
• Benign and Harmful Reward Errors in Language Model Post-Training — Paper by Razin et al. (2025) characterizing the effect of reward errors on optimization.

• Deep Reinforcement Learning from Human Preferences — Foundational work on using human preferences to train reward models.
• Training language models to follow instructions with human feedback — InstructGPT paper, a key example of RLHF in practice.

• Reward model accuracy and downstream task performance — Some works suggest that higher reward model accuracy generally leads to better downstream performance, which may seem to contradict the talk's claim that accuracy is not sufficient.