A Practical Field Guide to Optimizing Cost, Speed & Accuracy of LLMs | Niels Bantilan, Union.ai

Value of the Information & Strength of the Argument

The talk provides valuable, actionable insights for AI/ML teams considering SLM adoption. The argumentation is coherent and grounded in practical experience, using a concrete case study (SQL agent) to illustrate strategies. The speaker effectively addresses cost, speed, and accuracy trade-offs, and emphasizes the importance of orchestration and evaluation. However, the talk is largely based on anecdotal evidence and the speaker’s own experience, lacking rigorous empirical data or comparative studies. The argumentation is persuasive but not deeply technical, and some claims (e.g., 100x savings) are presented as optimistic estimates without detailed calculations.

Scientific Rigor, Source Quality, Title Accuracy

The talk references two key sources: Nvidia’s position paper on SLMs and the ‘Hidden Technical Debt in Machine Learning Systems’ paper. These are relevant and credible, but the talk does not provide detailed citations or URLs. The title accurately reflects the content, focusing on practical optimization strategies. The talk is well-structured and the speaker is knowledgeable, but the lack of explicit citations and the reliance on personal experience somewhat reduce the scientific rigor. The description includes a link to MLOps World, but no direct links to the cited papers.

The title accurately reflects the content: a practical guide to optimizing cost, speed, and accuracy of LLMs, with a focus on SLMs.

7/10

The talk is an expert opinion based on practical experience, referencing the Nvidia position paper and the hidden technical debt paper. It provides concrete strategies and examples, but lacks detailed citations and empirical validation.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and motivation: SLMs as the future of agentic AI, referencing Nvidia's position paper.
• 02:00Definition of SLMs and their advantages: sufficiency, operational suitability, and economic benefits.
• 05:00Barriers to SLM adoption: upfront investment, benchmark issues, and lack of awareness.
• 08:00Pain-driven development journey: from prototype to scaling issues, highlighting the need for evaluation and monitoring.
• 11:00Introduction of the SQL agent case study and the strategy overview.
• 14:00Identifying leverage points: swapping LLMs for SLMs in downstream tasks.
• 17:00Maintaining performance: AI unit tests, synthetic datasets, and LLM judges.
• 20:00Optimizing experimentation speed: prompt optimization and fine-tuning pipelines.
• 23:00Managing costs: heterogeneous compute and resource-aware orchestration.
• 26:00Optimizing inference speed: container reuse and auto-scaling.
• 29:00Conclusion: SLMs require investment in tools and practices; agent code is trivial compared to infrastructure.

• MLOps World — Conference website for the event where the talk was given.

• Nvidia Position Paper on SLMs — Referenced in the talk as the basis for the SLM future argument.
• Hidden Technical Debt in Machine Learning Systems — Referenced in the talk to illustrate the complexity of ML systems.