Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models

Item: Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models
Rating: 5
Author: PostSingularity Institute

by Afshine Amidi, Shervine Amidi · View the channel on PSI

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQFGenerative artificial intelligence

🎙 Afshine Amidi, Shervine Amidi 👥 1.3M 📅 October 9, 2026 ⏱ 103 min 👁 43K 📄 tutorial 🧭 2026-10-15

Available in: English (current) Français

Keywords

large language modelstransformer architecturemixture of expertsscaling lawsposition embeddings

Summary

This second lecture of Stanford’s CME295 course focuses on large language models (LLMs) and their relationship to the transformer architecture. It begins by recapping tokenization, embeddings, and the attention mechanism from the previous lecture, then introduces two major transformer variants: encoder-only models (BERT) and decoder-only models (GPT). BERT uses masked language modeling and next sentence prediction for pre-training, followed by fine-tuning for downstream tasks, while GPT frames every task as text-to-text generation through autoregressive decoding. The lecture discusses scaling laws, highlighting the trade-off between model size and training tokens, and the importance of compute-optimal training. It then delves into sparse Mixture of Experts (MoE) architectures, explaining how routers activate selective experts to increase model capacity without a proportional increase in computation. The session also covers practical architectural details such as position embeddings (including rotary position embeddings), sliding window attention, grouped query attention, normalization techniques (pre-norm vs. post-norm, LayerNorm vs. RMSNorm), and the next token prediction objective. Finally, it touches on prompting, context rot, and chain-of-thought reasoning, setting the stage for subsequent lectures.

172 words

Critical Evaluation

Value of the Information & Strength of the Argument

The lecture provides a comprehensive and well-structured overview of LLM architectures, systematically contrasting encoder-only, decoder-only, and encoder-decoder variants. It effectively explains the evolution from BERT to GPT, clarifying why decoder-only models became dominant. The argumentation is solid, consistently linking theoretical concepts to practical considerations, such as scaling laws and the compute-optimal balance between model parameters and training data. The discussion of MoE, including routing collapse and load balancing, is particularly valuable, as it addresses both the benefits and challenges of sparse architectures. The lecturer’s use of analogies and interactive questions reinforces understanding, making complex ideas accessible without oversimplification.

Scientific Rigor, Source Quality, Title Accuracy

The lecture demonstrates high scientific rigor, referencing seminal papers such as ‘Attention is All You Need’, BERT, GPT, and scaling laws papers, though specific citations are not always explicitly provided with references. The content aligns with current research consensus in the field. The title correctly indicates the lecture’s scope, which is exactly the topic of large language models within the course curriculum. The presentation quality is high, with clear diagrams and careful explanations of architectural nuances, contributing to the overall reliability of the information.

197 words

Title / Content Match

The title accurately reflects the lecture's focus on large language models within the transformer course context.

Quality & Reliability

9/10

Lecture from Stanford University's CME295 course delivered by experienced adjunct professors, covering established research and architectural concepts with high clarity and pedagogical structure.

Chapters

Cited Sources

CME295 Syllabus — Official course syllabus and schedule for CME295.
Stanford Graduate Education — Information about Stanford's graduate programs.
CME295 Playlist — YouTube playlist for the course, containing all lectures.

Concurring Sources

Attention is All You Need — Original transformer architecture paper, foundational to the lecture's discussion.
BERT: Pre-training of Deep Bidirectional Transformers — Paper introducing encoder-only models, directly relevant to the BERT section.
Scaling Laws for Neural Language Models — Seminal work on scaling laws, confirming the lecture's content on model scaling.
RoFormer: Enhanced Transformer with Rotary Position Embedding — Paper on rotary position embeddings, discussed in the lecture.

Contribution & Novelties

This lecture provides an excellent synthesis of LLM evolution, focusing on architectural decisions and practical trade-offs. It explains why decoder-only models prevail, clarifies the role of scaling laws in determining model size and training data, and offers a deep dive into MoE and modern attention mechanisms. The pedagogical approach, with interactive questions and relatable analogies, enhances understanding of complex topics.

Pour aller plus loin :

Attention is All You Need — The foundational transformer paper, directly relevant to the lecture’s core architecture discussion.
BERT: Pre-training of Deep Bidirectional Transformers — Original paper on encoder-only models, covered in the lecture.
Scaling Laws for Neural Language Models — Key reference for the scaling laws section, quantifying performance vs. model size and data.
RoFormer: Enhanced Transformer with Rotary Position Embedding — Paper introducing RoPE, a key positional encoding method discussed in the lecture.
Mixture of Experts — Wikipedia overview of MoE, matching the lecture’s coverage of sparse expert models.

156 words

Radar Profile

The radar profile shows a highly balanced performance with strong scores in information quantity, quality, and reliability, and a slightly lower but still robust technical level. This indicates a comprehensive and trustworthy educational resource that maintains high clarity without sacrificing depth.

Reliability 9/10