Lec 40: Graph Tracing, Batching & Concurrency

Lec 40: Graph Tracing, Batching & Concurrency

by Dr. Satyajit Das and Prof. Satyadhyan Chickerur · View the channel on PSI

Applied Sciences & Engineering Computing & Cybersecurity UYFComputer architecture and logic designUYFPParallel processing
🎙 Dr. Satyajit Das and Prof. Satyadhyan Chickerur 👥 229K 📅 September 10, 2026 ⏱ 28 min 👁 0 📄 tutorial 🧭 2026-09-10
Available in: English (current) Français

Keywords

graph tracingCUDA graphsdynamic batchingconcurrent servingtorch.compile

Summary

This lecture segment focuses on optimizing AI inference by using graph tracing, batching, and concurrency. It begins by contrasting eager execution with compiled graphs, explaining how operator call overhead (3-10 microseconds per call) accumulates to significant latency. Techniques such as torch.jit.trace, torch.jit.script, and torch.export are introduced for capturing graphs, each with trade-offs regarding control flow and dynamic shapes. The lecture then addresses CPU overhead in small-batch GPU inference, showing that kernel launch costs can dominate. CUDA graphs are presented as a solution to record and replay kernel launches, reducing overhead from ~20ms to ~1ms per step. Practical implementation via torch.compile with mode=‘reduce-overhead’ is discussed, along with limitations and workarounds for dynamic shapes. Batching strategies, including static and dynamic batching, are compared, highlighting continuous batching for token-level efficiency. Finally, concurrency in serving frameworks is covered, using async I/O to overlap preprocessing and inference, with examples from vLLM and FastAPI. The lecture concludes with notes on flash attention and operator fusion as complementary optimizations.

162 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture provides substantial value by systematically explaining how graph tracing and CUDA graphs reduce inference latency. It uses concrete numbers to illustrate overheads, making the argumentation compelling. The comparison of different tracing methods clarifies when to use each, and the demonstration of CUDA graph replay modes is practical. The argumentation is sound, though it leans on established practices rather than presenting novel results. The inclusion of batching and concurrency strategies completes the picture, making this a valuable resource for practitioners.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the content aligns with well-known optimization techniques used in production systems. The sources cited are limited to the course materials, but references to PyTorch, CUDA graphs, and vLLM are implicit. The title matches the content precisely. No external sources are directly cited, but the material is consistent with public documentation from NVIDIA and PyTorch. The lecture is technically accurate and well-structured.

162 words

Title / Content Match

Title accurately reflects the content: graph tracing, batching, and concurrency are the main topics covered.

Quality & Reliability

8/10

The content is presented by academic experts from IIT Guwahati, based on established practices in AI inference optimization. The lecture is well-structured, references standard tools and frameworks, and provides quantitative insights.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Contribution & Novelties

The lecture synthesizes key optimization techniques for LLM inference, focusing on graph tracing and CUDA graphs. It provides a clear workflow from torch.jit to torch.export, explaining trade-offs. The discussion on concurrency using async I/O is practically relevant. The content bridges the gap between theoretical knowledge and deployment considerations.

Pour aller plus loin :

90 words

Radar Profile

The profile shows balanced scores across all dimensions, indicating a well-rounded lecture with strong technical depth and reliable information. The highest scores are in information quality and technical level, suggesting the content is advanced and accurate, while the lower but still solid scores on quantity and global reliability reflect its focus on a specialized topic without extensive external references.

Reliability 8/10