The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior

The talk provides a compelling and well-structured argument for reconsidering how LLMs are evaluated. The speaker presents empirical evidence from a field study involving 800 users, which is a significant strength, as it moves beyond theoretical concerns to real-world data. The methodology is clearly explained: participants were recruited via Prolific, with balanced demographics (Black men, Black women, White men, White women), and asked to copy-paste questions into their own ChatGPT or Gemini interfaces. This approach captures the personalized context that offline evaluations miss. The comparison between offline and field responses reveals notable differences: field responses are more diverse, and for objective questions like MMLU, users sometimes receive answer choices never seen in offline runs. This is a critical finding, as it suggests that benchmark scores may not reflect the actual performance experienced by users. The introduction of ‘sock puppets’ as a scalable alternative is innovative, though the speaker acknowledges they do not fully replicate real-world diversity. The talk also touches on social implications, such as differential capabilities across users, which could exacerbate inequalities. However, the presentation is based on a limited set of questions (13 in the field study, 514 MMLU questions for sock puppets), and the results may not generalize broadly. The speaker does not delve into potential confounding factors, such as user behavior or interface differences, that might influence responses. Additionally, the talk is a research presentation, not a peer-reviewed publication, so the findings should be considered preliminary. The argumentation is logical and well-supported, but the lack of detailed statistical analysis in the talk leaves some questions unanswered. Overall, the talk is valuable for highlighting a critical gap in LLM evaluation and offers a practical direction for improvement.

The title accurately reflects the core argument: offline evaluations are inadequate because they ignore personalization, which significantly alters model behavior.

8/10

The talk presents empirical evidence from a field study with 800 users and compares offline evaluations with personalized interactions. The methodology is clearly described, and the results are statistically grounded. However, the talk is a presentation of ongoing research, and some details are not fully peer-reviewed yet.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction by Eve and start of Angelina Wang's talk.
• 01:00Definition of personalization and examples of chatbot implementations.
• 03:00Illustration of inconsistent memory behavior in Meta LLaMA across devices.
• 05:00Comparison of offline vs. field evaluations and description of the field study setup.
• 08:00Results showing higher diversity in field responses compared to offline.
• 10:00Introduction of sock puppets as a scalable evaluation method.
• 12:00Findings on MMLU questions: users get different answers, including unseen ones.
• 15:00Variation in scores across sock puppets and comparison to offline scores.
• 18:00Implications for benchmark leaderboards and individual user experiences.
• 20:00Discussion of social implications and future directions.

• Simons Institute Talk Page — Official page for the talk, providing abstract and speaker information.

• Simons Institute Talk Page — The talk's abstract aligns with the presented content.