Google Webinar | Risk Classification with LLMs

Value of the Information & Strength of the Argument

The value of the information is high for practitioners seeking to implement LLM-based classification in security. The speaker provides a concrete, actionable framework and shares real-world lessons learned, such as the need for a clear taxonomy, the importance of ground truth, and the benefits of temperature zero and thinking models. The argumentation is solid, based on practical experience rather than theoretical claims. The speaker acknowledges limitations and emphasizes that LLMs are not magic, which adds credibility. However, the presentation lacks quantitative evidence (e.g., specific accuracy numbers) and does not compare with alternative methods in depth.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is moderate. The speaker references the MITRE CWE-25 taxonomy and mentions internal Google practices but provides no formal citations or links to external sources. The title accurately reflects the content. The webinar is an expert opinion based on practical experience, not a peer-reviewed study. The lack of detailed data and external references reduces the overall scientific rigor, but the practical insights are valuable.

The title accurately reflects the content: a webinar on risk classification using LLMs.

7/10

The webinar provides a practical, experience-based overview of using LLMs for security risk classification. It includes concrete methodology, lessons learned, and acknowledges limitations. However, it lacks detailed quantitative results, formal citations, and is based on a single practitioner's perspective.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction by WiCyS board liaison and overview of the organization.
• 03:00Sharon introduces herself and the topic: using LLMs for risk classification.
• 06:00Definition of risk insights and why they are critical.
• 09:00Why LLMs are used: handling volume, inconsistency, and scaling.
• 12:00Multi-dimensional classification use cases: product name, high-risk categories, CWE-25, invariance, top risks.
• 15:00Step-by-step process: outline use case, develop prompt, collect ground truth, measure accuracy, iterate.
• 18:00Lessons learned on ground truth: expert alignment, sampling across classes, inter-rater agreement.
• 21:00Model selection: temperature zero, thinking models, multilabel classification, structured output.
• 24:00Prompt engineering: distinguishing trigger vs root cause, handling abstract classes, using examples.
• 27:00Precision and recall metrics and how to use them to tweak prompts.
• 30:00Conclusion: LLMs are more consistent and accurate than humans, but require clear taxonomy and ground truth.
• 33:00Q&A session begins: questions on determinism, multimodal models, model selection, data poisoning, open-source platforms.

• WiCyS Strategic Partner Webinars — Mentioned in the video description as a source for more webinars from WiCyS strategic partners.

• WiCyS Strategic Partner Webinars — The webinar is part of a series of webinars from WiCyS strategic partners, which may contain similar content.