[ИАД, весна 2026] Математические методы анализа текстов. Лекция 7: Training LLMs от 31.03.2026

Value of the Information & Strength of the Argument

The lecture provides a solid overview of the fundamental challenges and techniques in training LLMs. It uses concrete examples (LLaMA 3, DeepSeek V3) and quantitative memory calculations to illustrate the scale of the problem. The argumentation is logical, building from single-GPU limitations to distributed strategies. The explanation of mixed precision and its paradox (no memory savings) is clear. The presentation of gradient checkpointing and accumulation is practical, with trade-offs explained. The introduction to ZeRO is well-structured, showing how it addresses memory bottlenecks. However, the lecture is introductory and does not delve into advanced implementation details or recent research beyond the Hugging Face guide.

Scientific Rigor, Source Quality, Title Accuracy

The lecture is based on the Hugging Face LLM training guide, which is a reputable and practical resource. The presenter mentions this source and recommends it. However, no specific papers or primary sources are cited directly in the video. The content is accurate and aligns with current best practices. The title accurately reflects the content, focusing on training LLMs. The lecture is well-structured and technically sound, though it relies on secondary sources.

The title accurately reflects the content: a lecture on training large language models, covering memory optimization and parallelism techniques.

8/10

The lecture is based on the Hugging Face LLM training guide and covers established techniques (mixed precision, gradient checkpointing, gradient accumulation, ZeRO/FSDP). The presenter explains concepts clearly and provides quantitative examples. However, no primary sources are cited directly in the video, and the content is a summary of existing knowledge.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction: topic of the lecture, motivation, and overview of challenges (memory, compute, communication).
• 03:00Examples of large-scale training: LLaMA 3 (405B, 16k GPUs) and DeepSeek V3 (2k GPUs).
• 07:00Memory components during training: parameters, gradients, optimizer states, activations.
• 10:00Memory calculations for a 7B model in FP32: 112 GB for parameters, gradients, and optimizer states.
• 13:00Introduction to mixed precision training: FP16, BF16, and the master copy of weights.
• 16:00Explanation of loss scaling and why mixed precision does not reduce total memory but speeds up compute.
• 19:00Activation memory as the main bottleneck, especially with long sequences.
• 22:00Gradient checkpointing: full and selective, with trade-offs in compute and memory.
• 25:00Gradient accumulation: achieving large batch sizes with limited memory.
• 28:00Introduction to data parallelism and ZeRO stages (1-3), with FSDP as ZeRO-3.

• Hugging Face LLM training guide — The lecturer bases the lecture on this guide, which describes scaling experiments on thousands of GPUs.

• Hugging Face LLM training guide — The lecture directly follows this guide, which is a reputable source for practical LLM training.