Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson

Value of the Information & Strength of the Argument

The talk offers valuable insights from real-world experience, highlighting common pitfalls in ML deployment. Robertson’s argumentation is clear and well-structured, using a personal case study to illustrate the consequences of poor scoping. She provides actionable strategies, such as defining quantitative success metrics and involving stakeholders early. The emphasis on value over technical novelty is a strong point, and the discussion of model degradation and monitoring is practical. However, the arguments are largely anecdotal and lack empirical evidence or references to industry studies, which slightly weakens the overall rigor.

Scientific Rigor, Source Quality, Title Accuracy

The talk is based on the speaker’s professional experience, which lends credibility but also limits generalizability. No external sources are cited, and the only links provided are to NDC conferences. The title accurately reflects the content, and the talk stays on topic. The lack of citations is a minor weakness, but the practical nature of the content compensates. The speaker’s expertise is evident, and the advice is grounded in real-world scenarios.

The title accurately reflects the content, which focuses on reasons why ML models fail in production and strategies to mitigate these failures.

7/10

The talk is based on the speaker's 9 years of practical experience in ML deployment, providing concrete examples and actionable advice. However, it lacks formal citations or references to external research, and the evidence is anecdotal.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 02:00Introduction: Megan Robertson introduces herself and the topic of why great models fail.
• 05:00Reframing failures as learning opportunities and encouraging audience participation.
• 10:00Motivating example: social media analysis project that failed due to lack of stakeholder consultation and data cost.
• 15:00Introduction to two main reasons for model failure: scope not fully defined and world changes over time.
• 20:00Importance of proper scoping and overview of the scoping flowchart.
• 25:00Step 1: Identify stakeholders and end users, and define the problem to solve.
• 30:00Defining success with quantitative KPIs and common mistakes like building what is cool and evaluating on vibes.
• 35:00Step 2: Understand constraints and risks, including budget, bandwidth, and effects of bad predictions.
• 40:00Step 3: Identify possible solutions and the importance of aligning with business value.
• 45:00Step 4: Plan maintenance and monitoring, and strategies for model updating.

• NDC Conferences — Official NDC conference website, mentioned in the video description.
• NDC Toronto — Official NDC Toronto conference website, mentioned in the video description.

• Why Machine Learning Models Fail in Production — Article discussing common reasons for ML model failure, aligning with the talk's themes.

• The Myth of Model Interpretability — This paper argues that interpretability is often overrated, which contrasts with the talk's emphasis on stakeholder alignment and understanding.