But what is cross-entropy? | Compression is Intelligence Part 2

Item: But what is cross-entropy? | Compression is Intelligence Part 2
Rating: 5
Author: PostSingularity Institute

by 3Blue1Brown

Formal & Physical Sciences Science — General & History GPResearch and information: general GPFInformation theory

🎙 3Blue1Brown 👥 8.5M 📅 July 16, 2026 ⏱ 33 min 👁 526K 📄 science communication 🧭 2026-08-02

Available in: English (current) Français

Keywords

cross-entropycompressioninformation theorylanguage modelsKL divergence

Summary

This video from 3Blue1Brown explains the concept of cross-entropy, a fundamental measure in information theory, and its connection to compression and language model training. It begins with a motivating example from a 2002 paper that used gzip compression to cluster languages, illustrating how compression can reveal linguistic structure. The video then defines cross-entropy as the average number of bits needed to encode symbols from one distribution using a code optimized for another. It uses visual diagrams and simple examples to build intuition, highlighting that cross-entropy is minimized when the two distributions are identical, and that this minimum equals the entropy of the true distribution. The second half of the video applies these ideas to pre-training large language models, showing how the loss function used in next-token prediction is essentially cross-entropy. It also discusses distillation, where a smaller model is trained to mimic a larger one, and explains how cross-entropy relates to KL divergence. The video concludes by reframing language model training as a form of compression, emphasizing the deep connection between these concepts.

173 words

Critical Evaluation

The video is an exemplary piece of science communication, offering a rigorous yet accessible explanation of cross-entropy. It successfully bridges abstract mathematical concepts with practical applications in machine learning, making it valuable for both students and practitioners. The argumentation is solid: the video builds from first principles, using clear visualizations and concrete examples to illustrate each step. The connection between compression and language model training is well-established, and the explanation of why cross-entropy is the appropriate loss function is compelling. The sources cited, including the original ‘Language Trees and Zipping’ paper and Khan Academy materials on Lagrange multipliers, are credible and relevant. The video’s production quality is high, with animations that effectively convey complex ideas. One minor critique is that the video assumes some familiarity with probability and logarithms, but it does not require advanced knowledge. The title accurately reflects the content, and the video delivers on its promise to explain cross-entropy from the ground up. Overall, this is an excellent educational resource that provides deep insight into a core concept in information theory and machine learning.

177 words

Title / Content Match

The title accurately reflects the content, which focuses on explaining cross-entropy and its role in compression and language model training.

Quality & Reliability

9/10

The video is produced by 3Blue1Brown, known for rigorous mathematical explanations. It clearly defines cross-entropy, derives it from first principles, and connects it to practical applications in language models. The content is accurate and well-structured, with visual aids that enhance understanding. The sources cited are relevant and credible.

Chapters

Cited Sources

Language Trees and Zipping — Mentioned at the beginning as the motivating example for cross-entropy in compression.
Khan Academy: Lagrange multipliers and constrained optimization — Referenced for viewers wanting to learn more about Lagrange multipliers, which are relevant to the optimization discussed.
3Blue1Brown FAQ — Provides information about the manim library used for animations.
3Blue1Brown Home Page — Official website for the channel.
3Blue1Brown Mailing List — For updates on new videos.

Concurring Sources

Language Trees and Zipping — The paper's findings align with the video's explanation of using compression for language clustering.
Khan Academy: Lagrange multipliers — Provides background on optimization techniques relevant to the video's discussion.

External References

Contribution & Novelties

This video provides a clear and intuitive explanation of cross-entropy, connecting it to both compression and language model training. It offers a novel perspective by framing language model training as a form of compression, which is not commonly emphasized in standard machine learning courses. The visualizations and step-by-step derivations make the concept accessible to a wide audience.

Pour aller plus loin :

Information theory — Foundational concepts.
Cross entropy — Detailed mathematical treatment.
Kullback–Leibler divergence — Related measure of distribution difference.
Language Models are Unsupervised Multitask Learners — Discusses language model training.
The Annotated Transformer — For understanding transformer architectures.

100 words

Radar Profile

The radar profile shows high scores across all dimensions, indicating a well-rounded and reliable educational video. The strongest aspects are the quality and quantity of information, while the technical level is appropriately high but accessible.

Reliability 9/10

💬 Très positif. Sur les 30 commentaires analysés, les spectateurs expriment une admiration unanime pour la clarté pédagogique et la profondeur du contenu, avec de nombreux témoignages sur l'impact de la vidéo sur leur compréhension.