Stanford CS229 Machine Learning | Spring 2026 | Lecture 13: LLMs, Next-Word Prediction Loss

The lecture provides a solid introduction to representation learning and its application to large language models, with a clear pedagogical structure. The instructor begins with the concept of embeddings, which is fundamental to modern machine learning, and explains the motivation behind learning good representations. The discussion of supervised pretraining is concise but highlights the key limitation: the need for large labeled datasets, which are expensive to obtain. This sets the stage for the introduction of contrastive learning, a self-supervised alternative that leverages data augmentations to create positive pairs. The explanation of contrastive learning is clear, with the instructor emphasizing the need for negative pairs to avoid collapse, where all embeddings become identical. This is a critical point that is often overlooked in introductory materials. The lecture also addresses a student question about the ground truth of representations, clarifying that there is no single correct representation; the quality is judged by usefulness for downstream tasks. This is an important nuance that helps students understand the philosophy behind representation learning. The transition to large language models is logical, with the instructor introducing the next-word prediction loss as a powerful self-supervised objective. The explanation of how this loss enables learning from unlabeled text is accurate and aligns with current research. However, the lecture does not delve into the technical details of the transformer architecture or the specific training procedures used in modern LLMs, which may leave some students wanting more depth. The sources cited are limited to the course website and Stanford’s AI program page, which are appropriate for a course lecture but do not provide direct references to the research papers discussed. Overall, the lecture is rigorous and well-presented, but it is an introductory overview rather than a deep dive into the latest research. The adéquation between title and content is good, as the lecture does cover LLMs and the next-word prediction loss, though the initial focus on representation learning is a necessary foundation. The lecture’s strengths lie in its clear explanations and the instructor’s ability to address student questions effectively. The main weakness is the lack of specific citations to the literature, which would enhance the scientific rigor. Despite this, the content is accurate and up-to-date, making it a valuable resource for students.

The title accurately reflects the content: the lecture covers large language models and the next-word prediction loss, though the initial portion focuses on representation learning and contrastive learning as a foundation.

8/10

Lecture from Stanford CS229, taught by professors Chris Ré and Tengyu Ma. Content is technically rigorous, well-structured, and based on established machine learning concepts. The lecture is part of a reputable academic program, and the instructors are experts in the field. However, as a lecture, it presents established knowledge rather than new peer-reviewed research, and the specific claims are not individually sourced within the video.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to representation learning and embeddings
• 05:00Supervised pretraining for image representations
• 10:00Limitations of supervised pretraining and need for labels
• 15:00Introduction to contrastive learning and data augmentation
• 20:00Explanation of positive and negative pairs in contrastive learning
• 25:00Student question about ground truth representations
• 30:00Transition to large language models and next-word prediction loss
• 35:00Discussion of self-supervised learning for text and scaling
• 40:00Application of next-word prediction in modern LLMs
• 45:00Conclusion and summary of key concepts

• CS229 Course Website — Course materials and syllabus for CS229 Machine Learning, Spring 2026.
• Stanford AI Professional and Graduate Programs — Information about Stanford's AI programs, mentioned in the video description.

• CS229 Course Website — Official course page, consistent with the lecture content.