Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 3: Architectures
Keywords
Summary
114 words
Critical Evaluation
The lecture provides an excellent, up-to-date survey of transformer architectures, drawing on a wide range of recent models. The instructor’s approach of synthesizing common practices and variations is highly valuable for practitioners. The content is technically rigorous, with clear explanations of the motivations behind architectural choices, such as the shift to pre-norm for training stability and the adoption of SwiGLU and RoPE. The lecture is well-structured, starting with the vanilla transformer and progressively introducing modern modifications. The discussion of hyperparameters and training stability tricks is particularly useful. The instructor also highlights the trade-offs involved, such as the balance between model capacity and computational efficiency. The sources cited are primarily the models themselves (e.g., LLaMA, Qwen) and the course materials, which are appropriate for the context. The lecture’s main strength is its practical orientation, preparing students to implement and train their own models. However, it could benefit from more formal citations to specific papers for each architectural choice. Overall, this is an outstanding lecture that effectively bridges theory and practice.
169 words
Title / Content Match
The title accurately describes the content: a lecture on language model architectures, part of the CS336 course.
Quality & Reliability
9/10
Lecture by a Stanford professor, based on a survey of recent language model architectures, with references to specific models and papers. The content is technical and detailed, but the lecture format and lack of formal citations in the transcript reduce the score slightly.
Key Moments
Markers derived by PSI from the transcript: the creator did not define chapters.
- Introduction and course philosophy
- Overview of transformer variants and common choices
- Discussion of layer normalization placement (pre-norm vs post-norm)
- Activation functions: ReLU vs SwiGLU
- Positional encodings: sinusoidal vs RoPE
- Hyperparameters: FFN dimension, vocabulary size, etc.
- Training stability techniques and their connection to architecture
- Survey of recent models and their architectural choices
- Summary and key takeaways
Cited Sources
- CS336 Course Website — Course materials and syllabus
- CS336 Course Page on Stanford Online — Course enrollment information
- Stanford Online AI Programs — General information about Stanford's AI programs
- CS336 Course Playlist — Playlist of course lectures
Concurring Sources
- Attention Is All You Need — Original transformer architecture, referenced as the starting point.
- LLaMA: Open and Efficient Foundation Language Models — LLaMA model, which popularized pre-norm, SwiGLU, and RoPE.
Contribution & Novelties
This lecture provides a comprehensive and current survey of transformer architectures, synthesizing the design choices made in recent language models. It offers practical guidance for implementing and training models, emphasizing the importance of architectural decisions for stability and performance. The lecture also highlights the trade-offs involved and the rationale behind common practices.
Pour aller plus loin :
- Attention Is All You Need — The original transformer paper, foundational for understanding the architecture.
- RoFormer: Enhanced Transformer with Rotary Position Embedding — Introduces RoPE, a key positional encoding method discussed in the lecture.
- GLU Variants Improve Transformer — Introduces SwiGLU, the activation function used in modern models.
- LLaMA: Open and Efficient Foundation Language Models — The LLaMA model, which influenced many architectural choices.
121 words
Radar Profile
The radar profile shows high scores across all dimensions, indicating a lecture that is both informative and technically deep. The balance between quantity and quality of information is particularly strong, with a slight emphasis on technical depth.