Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 5: GPUs, TPUs

The lecture provides a comprehensive and rigorous introduction to GPU computing for deep learning, particularly in the context of large language models. The instructor, Tatsunori Hashimoto, demonstrates deep expertise and a clear pedagogical approach, building concepts progressively from hardware fundamentals to advanced optimization techniques. The content is highly accurate and up-to-date, reflecting current best practices in GPU programming and model optimization. The explanation of the memory hierarchy is particularly strong, emphasizing the critical role of memory bandwidth and latency in determining performance. The six optimization tricks are well-chosen and clearly explained, with practical examples that illustrate their impact. The final case study on FlashAttention effectively synthesizes the material, showing how the concepts can be applied to a real-world algorithm. The lecture also appropriately references external resources, such as the GPU Mode community and the TPU/GPU book, which are reputable and valuable for further study. The presentation is well-structured, with clear sections and a logical flow. The use of diagrams and plots enhances understanding, though some may be complex for beginners. The lecture does not include any obvious biases or unsupported claims; it is grounded in established knowledge and recent research. The main limitation is that it does not cover TPUs in depth, despite the title mentioning them, but this is a minor omission given the focus on GPUs. Overall, this is an excellent educational resource that would benefit students, researchers, and practitioners seeking to understand the hardware foundations of efficient deep learning.

The title accurately reflects the content: a lecture on GPUs and TPUs as part of a language modeling course.

9/10

Lecture from Stanford University by a professor, covering established hardware concepts and recent optimizations like FlashAttention. Content is technically accurate and well-structured, with references to reputable resources.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to the systems portion of the course, focusing on GPUs, parallelization, and inference.
• 03:00Overview of the lecture structure: hardware model, six optimization tricks, and FlashAttention.
• 05:00Discussion on the importance of compute scaling and the role of GPUs in enabling larger models.
• 10:00Comparison of CPU and GPU design philosophies: latency vs. throughput.
• 15:00Explanation of GPU architecture: streaming multiprocessors (SMs), memory hierarchy, and physical layout.
• 20:00Introduction to the six optimization tricks for GPU performance.
• 30:00Detailed explanation of kernel fusion and its benefits.
• 40:00Discussion on memory coalescing and its impact on memory bandwidth.
• 50:00Explanation of tensor cores and how to leverage them for matrix multiplication.
• 1:00:00Introduction to FlashAttention and how it applies the previously discussed techniques.
• 1:10:00Detailed walkthrough of FlashAttention's memory optimization and its performance benefits.

• CS336 Course Website — Course syllabus, schedule, and materials.
• Stanford Online CS336 Course Page — Information about enrolling in the course.
• Stanford AI Programs — Overview of Stanford's online AI programs.
• Course Playlist — Playlist of all lectures for the course.

• GPU Mode — Community and resources for GPU programming, mentioned in the lecture.
• TPU/GPU Book — Book by Google researchers covering TPU and GPU architecture, mentioned in the lecture.