Jürgen Schmidhuber on Robotics, World Models, and NNAISENSE (Oral History Pt. 2)

The interview offers a valuable first-hand account of the development of recurrent neural networks and their theoretical advantages over transformers. Schmidhuber’s arguments are technically sound, drawing on well-established concepts in computation theory and his own published research. He effectively illustrates the limitations of transformers in tasks requiring systematic generalization, such as parity and context-free language recognition, and provides concrete examples. The historical context he provides, including his early work on attention and self-supervised pre-training, is a significant contribution to the understanding of AI’s evolution. However, the discussion is largely based on his personal opinions and retrospective interpretations, which may be subject to bias. While he references his own papers, he does not provide external validation or discuss potential counterarguments in depth. The interview also touches on speculative topics like self-replicating robots and galaxy exploration, which are more visionary than empirically grounded. Overall, the content is informative and thought-provoking, but it should be viewed as an expert’s perspective rather than a comprehensive, balanced review. The title accurately reflects the content, and the interview maintains a high level of technical rigor throughout.

The title accurately reflects the content, focusing on robotics, world models, and NNAISENSE, with a broader discussion of recurrent networks and AI history.

8/10

Interview with a leading AI researcher, providing historical context and technical insights. Claims are generally well-supported by references to his own published work, though some statements are opinionated and not peer-reviewed in this context.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction and question about LSTMs vs transformers.
• 02:00Discussion on linear vs quadratic scaling of RNNs and transformers.
• 05:00Explanation of recurrent networks as general-purpose computers, with examples of context-free languages.
• 10:00Parity problem example illustrating the difficulty for transformers.
• 15:00Historical overview of attention mechanisms, referencing 1991 and 1993 work.
• 20:00Discussion on self-supervised pre-training in 1991 and its resurgence.
• 30:00Introduction to NNAISENSE and its focus on robotics and physical world AI.
• 40:00Industrial applications, including 3D printing of metals.
• 50:00Belief in world models and recurrent networks for robotics.
• 1:00:00Vision of self-replicating robots exploring the universe.

• Computer History Museum Oral History Collection — Transcript and additional information about the interview.

• LSTM paper (Hochreiter & Schmidhuber, 1997) — Original paper introducing LSTM, supporting claims about recurrent network capabilities.
• World Models (Ha & Schmidhuber, 2018) — Paper on world models, aligning with Schmidhuber's discussion of world models in robotics.

• Attention Is All You Need (Vaswani et al., 2017) — The transformer paper, which Schmidhuber criticizes for its limitations, but which has been highly influential and successful in practice.