Introduction to LLM Inference

Introduction to LLM Inference

by San Diego Machine Learning

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence
🎙 San Diego Machine Learning 👥 21K 📅 March 21, 2026 ⏱ 90 min 👁 1K 📄 tutorial 🧭 2026-08-16
Available in: English (current) Français

Keywords

LLM inferencetransformer architecturedecoder-onlypre-normresidual streamtoken generationGPU utilizationinference optimization

Summary

This meetup presentation introduces a new book club series on LLM inference. The speaker, Ted, shares his journey from a specific use case to writing a book on the topic. He begins by explaining the fundamental process of autoregressive text generation, where the model predicts one token at a time, appending each to the prompt. This contrasts with one-shot models like image classifiers, highlighting the unique challenges of LLM inference. He then dives into the transformer architecture, focusing on the decoder-only structure used in modern LLMs. He clarifies that the original encoder-decoder transformer is not what production models use. He explains the pre-norm formulation, where normalization is applied before each sublayer, and emphasizes the residual stream that carries the input through the network. He clarifies that the ‘N’ in the architecture diagram represents N separate layers with distinct weights, not N passes through the same layer. He also explains that multiple tokens are processed simultaneously by adding a batch dimension to the input tensor, not by creating multiple copies of the network. The presentation aims to provide the right intuition for understanding inference bottlenecks and optimization techniques.

187 words

Critical Evaluation

Value of the Information & Strength of the Argument

The presentation provides valuable insights into the fundamentals of LLM inference, particularly the architectural details that are often glossed over. The speaker’s explanation of the residual stream and pre-norm formulation is clear and helps build a mental model for understanding inference challenges. The argumentation is solid, based on practical experience and a deep understanding of the subject. The speaker effectively motivates the need for optimization by contrasting LLM generation with one-shot models. The use of diagrams and analogies (e.g., the statue analogy) aids comprehension. The discussion of the physical vs. virtual nature of token processing is particularly insightful for understanding GPU utilization.

Scientific Rigor, Source Quality, Title Accuracy

The presentation demonstrates scientific rigor by accurately describing the transformer architecture and inference process. The speaker references the original ‘Attention is All You Need’ paper and mentions current research like ‘Attention Residuals’. However, the talk is based on the speaker’s own book, which is a work in progress and not peer-reviewed. The sources cited are primarily the book’s GitHub repository and a Slack community link, which are not academic references. The title accurately reflects the content, providing a solid introduction to LLM inference. The speaker’s disclosure about using LLMs for writing is transparent, but the technical content appears reliable.

216 words

Title / Content Match

The title accurately reflects the content, which provides a foundational overview of LLM inference, including architecture and optimization challenges.

Quality & Reliability

7/10

The presentation is technically accurate and well-structured, drawing on established knowledge of transformer architectures and inference optimization. The speaker demonstrates deep understanding and provides clear explanations. However, the content is based on personal research and experience, not peer-reviewed, and the book is a work in progress. The use of LLMs for writing is disclosed, but the core technical content appears sound.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

  • San Diego Machine Learning Book Club GitHub — Repository containing notes, slides, and videos of prior meetups, including the book being discussed.
  • SDML Slack Community — Community for discussion and questions about ML, used for sharing links and password for the online meetup.

Concurring Sources

Contribution & Novelties

The presentation offers a fresh perspective on LLM inference by focusing on the data flow and physical constraints of GPU execution, rather than just the mathematical operations. It clarifies common misconceptions about the transformer architecture, such as the role of the residual stream and the pre-norm formulation. The speaker’s practical experience and the book format provide a structured approach to understanding inference optimization.

Pour aller plus loin :

  • Attention Is All You Need — The original transformer paper, foundational for understanding the architecture.
  • The Illustrated Transformer — A visual guide to the transformer, useful for beginners.
  • vLLM — A popular inference framework that implements many optimizations discussed in the talk.
  • SGLang — Another inference framework with advanced features.
  • Attention Residuals — Recent paper on modifying the residual stream, mentioned in the talk (URL uncertain, so cited by name only).

139 words

Radar Profile

The radar profile shows high scores in information quantity and quality, indicating a content-rich and accurate presentation. The technical level is moderately high, suitable for an audience with some ML background. The overall reliability is good, though not perfect due to the informal setting and lack of peer review.

Reliability 7/10