He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]

Item: He Co-Invented the Transformer. Now: Continuous Thought Machines [Llion Jones / Luke Darlow]
Rating: 5
Author: PostSingularity Institute

by Machine Learning Street Talk

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQNNeural networks and fuzzy systems

🎙 Machine Learning Street Talk 👥 218K 📅 November 23, 2025 ⏱ 72 min 👁 107K 📄 expert opinion 🧭 2026-08-15

Available in: English (current) Français

Keywords

TransformerContinuous Thought Machineadaptive computationAI research culturespiral representation

Summary

In this interview, Llion Jones, co-inventor of the Transformer, and Luke Darlow from Sakana AI discuss the limitations of current AI architectures and introduce their new model, the Continuous Thought Machine (CTM). They argue that the industry is stuck in a local minimum, focusing on incremental tweaks to Transformers rather than exploring fundamentally different approaches. Using the analogy of solving a spiral, they illustrate how neural networks often ‘fake’ understanding without true conceptual representation. The CTM, inspired by biological neurons and synchronization, offers native adaptive computation, allowing the model to ’think’ longer on harder problems and correct its own mistakes. The conversation also covers the importance of research freedom, the pressures of commercialization, and the potential for AI to assist in scientific discovery. The discussion is technical yet accessible, providing a compelling case for exploring alternative architectures beyond the Transformer paradigm.

141 words

Critical Evaluation

Value of the Information & Strength of the Argument

The value of this discussion lies in its unique perspective from a co-inventor of the Transformer, offering critical insights into the current state of AI research. The argumentation is strong, supported by concrete examples like the spiral problem and references to relevant papers (e.g., ‘Intelligent Matrix Exponentiation’). The hosts and guests engage in a thoughtful dialogue, exploring both the technical and cultural aspects of AI development. The introduction of the CTM is well-motivated, with clear explanations of its advantages over Transformers, such as adaptive computation and better calibration. The discussion is balanced, acknowledging the strengths of Transformers while highlighting their limitations. Overall, the content is highly valuable for anyone interested in the future of AI architectures.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high, with guests referencing specific papers and providing technical details. The sources cited are credible, including arXiv papers and research from Sakana AI. The title accurately reflects the content, focusing on the co-inventor’s perspective and the introduction of the CTM. The discussion is well-structured, with clear explanations and minimal speculation. The hosts also bring in relevant external references, such as Kenneth Stanley’s book and Sara Hooker’s ‘Hardware Lottery’, enriching the context. The adéquation between title and content is excellent, as the video delivers exactly what it promises: an in-depth conversation about moving beyond Transformers.

229 words

Title / Content Match

The title accurately reflects the content: it highlights Llion Jones's role as co-inventor of the Transformer and introduces the Continuous Thought Machine as the main topic.

Quality & Reliability

8/10

High-quality discussion with two leading AI researchers, providing deep insights into the limitations of current architectures and introducing a novel approach (CTM). The claims are supported by references to specific papers and the discussion is technically rigorous, though it remains an opinion/interview rather than a peer-reviewed study.

Chapters

Cited Sources

Continuous Thought Machines — The paper introducing the CTM architecture, discussed in detail.
The Hardware Lottery — Referenced in the discussion about the dominance of certain architectures.
Intelligent Matrix Exponentiation — Used as an example of a model that represents a spiral as a spiral.
Why Greatness Cannot be Planned — Book by Kenneth Stanley, influential on Sakana AI's philosophy.
Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis — Referenced in the discussion about shortcut learning and representation issues.
A Spline Theory of Deep Networks — Mentioned in relation to the piecewise linear nature of ReLU networks.
On the Biology of a Large Language Model — Referenced in the discussion about biological inspiration.
Neural Turing Machine — Mentioned as a precursor to adaptive computation ideas.
Adaptive Computation Time for Recurrent Neural Networks — Referenced in the context of adaptive computation.
Sudoku Bench — Benchmark used to measure reasoning capabilities.
Sakana AI — Company website, mentioned as the research lab.
CTM page on Sakana AI — Additional information about the CTM.

Concurring Sources

The Hardware Lottery — Supports the idea that architectural dominance can be due to hardware and ecosystem factors.
Why Greatness Cannot be Planned — Aligns with the discussion on research freedom and open-ended exploration.
Intelligent Matrix Exponentiation — Provides an example of a model that represents a spiral as a spiral, supporting the argument for better representations.

External References

Contribution & Novelties

This video provides a unique insider perspective on the limitations of Transformers and introduces a novel architecture (CTM) that addresses key shortcomings. The discussion goes beyond technical details to explore the cultural and structural factors that hinder AI research innovation. The CTM’s approach to adaptive computation and synchronization offers a fresh direction for future research.

Pour aller plus loin :

Continuous Thought Machines (arXiv) — The original paper, essential for understanding the technical details.
The Hardware Lottery (arXiv) — Discusses how hardware and ecosystem factors can trap AI research in local minima.
Why Greatness Cannot be Planned (Amazon) — Kenneth Stanley’s book on the importance of open-ended exploration in research.
Intelligent Matrix Exponentiation (arXiv) — The paper with the spiral example, illustrating representation issues.
A Spline Theory of Deep Networks (PMLR) — Provides theoretical background on the piecewise linear nature of ReLU networks.
On the Biology of a Large Language Model (Transformer Circuits) — Explores biological analogies in LLMs, relevant to the CTM’s inspiration.

164 words

Radar Profile

The radar profile shows high scores across all dimensions, indicating a well-rounded and informative discussion. The video excels in providing substantial information and maintaining high quality, with a strong technical depth that is balanced by accessible explanations. The reliability is high due to the credibility of the speakers and the references provided.

Reliability 8/10

💬 Très positif. Sur les 30 commentaires analysés, les spectateurs expriment un enthousiasme marqué pour la profondeur de la discussion et la qualité des intervenants, certains soulignant l'importance de la liberté de recherche et la pertinence des idées présentées.