Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 4: Attention Alternatives

The lecture provides a comprehensive and rigorous overview of advanced attention mechanisms and mixture of experts, two critical areas in modern language model design. The instructors, Percy Liang and Tatsunori Hashimoto, are leading researchers in the field, lending high credibility to the content. The presentation is well-structured, starting with the motivation for long context and the computational challenges, then systematically introducing linear attention as a solution. The explanation of the associativity of multiplication and the equivalence to RNNs is particularly clear, making complex concepts accessible. The lecture also grounds theoretical ideas in practical implementations, citing models like Minimax M1 and Mamba 2, which demonstrates real-world applicability. The discussion of mixture of experts is thorough, covering routing mechanisms, load balancing, and expert parallelism, which are essential for scaling these models. The lecture does not shy away from technical details, such as the trade-offs between different approaches and the importance of systems optimizations like Flash Attention. The content is up-to-date, reflecting the latest research and industry trends. The only minor limitation is that the lecture assumes prior knowledge of transformer architectures, but this is appropriate for a course at this level. Overall, the lecture is an excellent resource for anyone seeking a deep understanding of these advanced topics.

The title accurately reflects the content, which focuses on alternatives to standard attention mechanisms for language models.

9/10

Lecture from Stanford University by leading AI professors, covering established and emerging research in attention mechanisms. Content is technically rigorous, well-structured, and based on published research and industry implementations. The lecture is part of a formal course, ensuring high academic standards.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and overview of the lecture topics: attention alternatives and mixture of experts.
• 03:00Motivation for longer context lengths and the quadratic cost of attention.
• 10:00Introduction to linear attention via associativity of matrix multiplication.
• 20:00Equivalence of linear attention to RNNs and benefits for inference.
• 30:00Example of linear attention in practice: Minimax M1 hybrid model.
• 40:00Enhancements to linear attention: gating and state space models like Mamba 2.
• 50:00Introduction to mixture of experts (MoE) and its benefits.
• 1:00:00Routing mechanisms and load balancing in MoE.
• 1:10:00Expert parallelism and practical considerations for training MoE models.
• 1:20:00Conclusion and summary of key takeaways.

• CS336 Course Website — Course materials, syllabus, and lecture slides.
• CS336 Course Enrollment — Information about enrolling in the online course.
• Stanford Online AI Programs — Overview of Stanford's online AI programs.
• Course Playlist — Playlist of all lectures for the course.

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces — The lecture discusses Mamba 2 as an enhancement of linear attention, aligning with this paper.
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — The lecture emphasizes the importance of systems optimizations like FlashAttention, which is described in this paper.

• Attention Is All You Need — The original transformer paper proposes full quadratic attention, while the lecture argues for alternatives to reduce computational cost.