CS 471/671 (How to grow language models) - #13 - Fall 2026

Value of the Information & Strength of the Argument

The lecture provides valuable, up-to-date information on best practices for training transformer language models. The instructor substantiates claims with empirical plots and logical reasoning, such as explaining gradient imbalance in post-norm architectures. He also distinguishes between theoretical FLOPs and runtime overhead, offering practical insights for implementation. Arguments are well-structured, building from architectural families to specific normalization choices, and are supported by references to common practices in the field.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is scientifically rigorous, drawing on established research and empirical findings presented in a university setting. The instructor mentions specific papers and sources (e.g., the course page) but does not provide full citations in the video; the linked course page likely contains references. The title accurately matches the content, which focuses on scaling and growing language models. The pedagogical approach is clear and well-organized, ensuring technical accuracy without oversimplification.

The title 'How to grow language models' accurately reflects the lecture's focus on model scaling practices and architectural choices, including normalization and attention variations.

8/10

Lecture from a recognized university program, covering established and recent research in transformer architectures. The instructor references consistent empirical results and provides structured explanations, though the content is pedagogical rather than primary research.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 01:25Final project overview: guidelines for proposals, reports, and poster session.
• 05:00Project proposal structure: motivation, hypothesis, experiments, datasets, expected outcomes.
• 10:00Introduction to transformer architecture families: decoder-only, encoder-decoder, encoder-only.
• 15:00Discussion of normalization conventions: post-norm vs pre-norm, with empirical gradient plots.
• 20:00Introduction of RMSNorm and analysis of computational efficiency vs runtime on GPUs.

• Course page for CS 471/671 — Course materials, examples of proposals, and additional resources are provided on this page.

• Course page for CS 471/671 — Provides course materials and potential references supporting the lecture content.