CS 471/671 (How to grow language models) - #9 - Fall 2026

Value of the Information & Strength of the Argument

VALUE: The lecture provides a solid, practical understanding of tokenization, an often-underappreciated component of language models. Using the OpenAI tokenizer interactively gives concrete insights into real-world behavior, while the hand-coded BPE example clarifies the training mechanism in a way that abstract descriptions rarely achieve. ARGUMENTATION: The argumentation is logical and incremental: it starts by motivating subword tokenization via the drawbacks of character and word levels, then decomposes the pipeline into clearly separated stages, and finally drills down into the BPE algorithm with a traceable example. The instructor addresses student questions on implementation details, such as merging rules and space handling, reinforcing the reasoning behind design choices.

Scientific Rigor, Source Quality, Title Accuracy

SCIENTIFIC RIGOR: The content is based on established algorithms (BPE) and standard NLP practices. The instructor accurately explains trade-offs and openly discusses limitations, such as normalization costs and the arbitrary nature of token boundaries. SOURCES: The only cited external resource is the official course page, which is appropriate. The lecture relies on well-known examples (OpenAI tokenizer) and does not introduce unsupported claims. ADEQUATION TITRE: The title indicates a course on growing language models, and this session on tokenization is a fundamental part of that process; the lecture matches the expected scope, though it is a specific subtopic rather than a broad overview.

The title suggests a course on growing language models; this lecture specifically addresses tokenization, a core component of building language models, so the content aligns well with the course theme.

9/10

University lecture by a reputable institution (JHU CLSP) covering established algorithms (BPE) with clear demonstrations and accurate technical details.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Recap of tokenization trade-offs and introduction to subword tokenization.
• 05:00Live demo of OpenAI's tokenizer on sample text, showing token splitting and special handling of numbers and spaces.
• 15:00Detailed explanation of the tokenization pipeline: normalization (NFKC), pre-tokenization into chunks, and post-processing steps.
• 25:00Step-by-step walkthrough of BPE training on a small corpus, including pair counting and merging iteration.
• 40:00Discussion of scalability, efficient data structures for pair counting, and handling of spaces in BPE; Q&A session.

• CS 471/671 Fall 2026 Course Page — Official course page providing syllabus, materials, and context for the lecture series.