Scheduling Impacts on LLM Inference

Item: Scheduling Impacts on LLM Inference
Rating: 4
Author: PostSingularity Institute

by San Diego Machine Learning

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence

🎙 San Diego Machine Learning 👥 21K 📅 May 20, 2026 ⏱ 86 min 👁 139 📄 expert opinion 🧭 2026-08-16

Available in: English (current) Français

Keywords

continuous batchingdynamic batchingchunked prefillKV cachetime-to-first-token

Summary

This meetup presentation, part of a book club series on LLM inference, focuses on scheduling techniques for serving LLMs. The speaker begins by contrasting the prefill phase (compute-heavy, processes entire prompt) with the decode phase (memory-bandwidth-bound, generates tokens one at a time). He explains the importance of batching to amortize the cost of loading large model weights, deriving that arithmetic intensity equals batch size. He then critiques static batching (from training code) for head-of-line blocking and poor GPU utilization. Dynamic batching introduces a timeout to avoid waiting for a full batch, but still processes requests in a request-centric manner. The main improvement is continuous batching (from the Orca paper), which schedules at the iteration level, allowing new requests to be inserted between decode steps, significantly improving GPU utilization and time-to-first-token, though it introduces bubbles that hurt time-per-output-token. To mitigate this, chunked prefill splits large prefills into smaller chunks, allowing interleaving with decode steps and reducing wait times. The discussion includes audience questions about implementation and cost optimization, and the speaker references the book ‘LLM Inference Illustrated’ and the Orca paper.

180 words

Critical Evaluation

Value of the Information & Strength of the Argument

The presentation provides valuable insights into LLM serving optimization, clearly explaining the trade-offs between different batching strategies. The argumentation is solid, building from first principles (arithmetic intensity) to practical scheduling techniques. The speaker effectively uses diagrams and examples to illustrate concepts like head-of-line blocking and the impact of prefill on decode latency. The discussion is technically sound and addresses real-world concerns such as GPU utilization and cost efficiency.

Scientific Rigor, Source Quality, Title Accuracy

The presentation is scientifically rigorous in its technical explanations, but it lacks formal citations. The speaker references the Orca paper (2022) and the book ‘LLM Inference Illustrated’, which is available online. The title accurately reflects the content. No comments were provided for analysis.

126 words

Title / Content Match

The title accurately reflects the content, which focuses on scheduling techniques for LLM inference.

Quality & Reliability

7/10

The presentation is technically accurate and grounded in established concepts (prefill/decode, batching, KV cache), but it is an informal meetup discussion without formal citations or peer review. The speaker demonstrates deep understanding, but the content is not rigorously sourced.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

LLM Inference Illustrated — The book being discussed, specifically chapter 5 on scheduling.
SDML GitHub repository — Contains notes and slides for the book club series.
SDML Slack community — Community for discussion and meeting password.

Concurring Sources

Orca: A Distributed Serving System for Transformer-Based Generative Models — The paper that introduced continuous batching, which the speaker references.

Contribution & Novelties

The presentation offers a clear, practical overview of scheduling techniques for LLM inference, emphasizing the trade-offs between prefill and decode. It provides a valuable synthesis of concepts like continuous batching and chunked prefill, making them accessible to a technical audience. The discussion of arithmetic intensity and its relation to batch size is particularly illuminating.

Pour aller plus loin :

Orca: A Distributed Serving System for Transformer-Based Generative Models — The seminal paper on continuous batching.
vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — A popular open-source LLM serving system that implements continuous batching and paged KV cache.
Tensor Parallelism — A technique for distributing model weights across GPUs, relevant to scaling inference.

114 words

Radar Profile

The radar profile shows high scores in information quantity and technical level, with slightly lower scores in information quality and reliability, reflecting the informal yet technically deep nature of the presentation.

Reliability 6/10