Ying Fan: Looped Transformers for Length Generalization

Item: Ying Fan: Looped Transformers for Length Generalization
Rating: 4
Author: PostSingularity Institute

by Ying Fan

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQNNeural networks and fuzzy systems

🎙 Ying Fan 👥 3K 📅 October 3, 2025 ⏱ 33 min 👁 241 📄 expert opinion 🧭 2026-08-16

Available in: English (current) Français

Keywords

length generalizationlooped transformersNRASPRASPnext token prediction

Summary

Ying Fan presents her work on using looped transformers to achieve length generalization in algorithmic tasks. She begins by framing length generalization as a specific case of easy-to-hard generalization, citing examples from graphs, mazes, and matrices. She then highlights the poor length generalization of standard transformers trained with next-token prediction on tasks like addition and multiplication. After reviewing existing solutions that are task-specific (e.g., reversing output, positional embeddings, chain-of-thought), she proposes a more general approach based on two principles: decomposition and reuse. She introduces the NRASP framework, which extends RASP to allow a number of steps that depends on input length. She argues that many tasks (copy, parity, addition) have NRASP solutions that can be implemented with a looped transformer. The proposed architecture is a decoder-only transformer without positional embeddings, with input injection, and trained end-to-end with a step count as side information. At inference, the model can either use a known step count or an adaptive stopping criterion based on confidence. Experimental results on copy, parity, and addition show that looped transformers significantly outperform next-token prediction baselines, including those with extra compute. The talk concludes with a discussion of related work and open questions.

195 words

Critical Evaluation

Value of the Information & Strength of the Argument

The talk provides valuable insights into why standard transformers fail at length generalization and proposes a principled solution. The argumentation is strong: it builds on prior work (RASP, looped architectures) and provides a clear theoretical framework (NRASP) that explains why the proposed method should work. The experimental results, while on toy tasks, convincingly demonstrate the superiority of looped transformers over next-token prediction baselines. The speaker also discusses limitations and open questions, such as the convergence behavior for parity, which adds to the credibility.

Scientific Rigor, Source Quality, Title Accuracy

The talk is scientifically rigorous, referencing the paper on arXiv (2409.15647) and building on established concepts like RASP and looped transformers. The sources are appropriate and credible. The title accurately reflects the content. The presentation is clear and well-structured, with a logical flow from problem setup to solution to results. No comments were provided, so no analysis of public reception is included.

160 words

Title / Content Match

The title accurately reflects the content: the talk focuses on using looped transformers to improve length generalization.

Quality & Reliability

8/10

The talk presents a novel framework (NRASP) and experimental results from a peer-reviewed paper (arXiv:2409.15647). The speaker is a researcher at Microsoft Research with a PhD, and the content is technically rigorous. However, the presentation is a summary and may omit some details, and the results are on toy tasks, limiting generalizability.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Looped Transformers as Programmable Computers — The paper presented in the talk, introducing the NRASP framework and looped transformer architecture.

Concurring Sources

RASP: A Programming Language for Neural Networks — The paper defines RASP, which the talk extends to NRASP, and shows limitations of standard transformers.
Universal Transformers — A looped transformer architecture that the proposed model is a special case of.

Dissenting Sources

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — This work suggests that chain-of-thought can improve length generalization, but the talk argues that such task-specific approaches are less general than the proposed looped transformer method.

Contribution & Novelties

The talk introduces a novel framework (NRASP) that formalizes a class of algorithms solvable by looped transformers, and demonstrates empirically that looped transformers can achieve length generalization on tasks where standard transformers fail. The key innovation is the combination of decomposition and reuse, which allows the model to adapt its computation depth to the input length, without requiring task-specific modifications.

Pour aller plus loin :

RASP: A Programming Language for Neural Networks — The basis for NRASP, providing a formal model of transformer computation.
Universal Transformers — A looped transformer architecture that inspired the proposed model.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Discusses a related approach to improving reasoning and length generalization.

115 words

Radar Profile

The radar profile shows high scores across all dimensions, indicating a well-rounded and technically strong presentation. The talk excels in providing novel insights and rigorous experimental validation, with a slight emphasis on technical depth.

Reliability 8/10