Yash Sarrof: Length Generalization of Transformers with a Growing Test time Alphabet
Keywords
Summary
131 words
Critical Evaluation
Value of the Information & Strength of the Argument
The talk provides valuable insights into the theoretical foundations of transformer generalization, extending prior work to a more realistic scenario where vocabulary size is not fixed. The argumentation is solid, building on established concepts like C-RASP and limit transformers, and clearly motivates the need for symbolic generalization. The speaker effectively uses examples from planning domains to illustrate the concepts. The proof sketch is coherent, though high-level, and the speaker acknowledges limitations, such as the removal of positional predicates for technical reasons.
Scientific Rigor, Source Quality, Title Accuracy
The talk demonstrates scientific rigor by referencing relevant literature, including the original RASP paper, C-RASP, and the work of Huang et al. The speaker clearly distinguishes between empirical observations and theoretical results. The title accurately reflects the content, focusing on length generalization with a growing alphabet. The sources cited are appropriate and directly related to the topic. The talk is a presentation of original research, and the speaker is transparent about the scope and limitations.
171 words
Title / Content Match
The title accurately reflects the content: the talk focuses on length generalization of transformers, specifically addressing the novel aspect of a growing test-time alphabet.
Quality & Reliability
8/10
The talk presents original research with a formal proof sketch, building on established theoretical frameworks (C-RASP, limit transformers). The speaker is a PhD student at a recognized institution, and the work is accepted at ICML. The presentation is rigorous, with clear definitions and a structured argument. However, the talk is a seminar presentation, not a peer-reviewed paper, and the proof sketch is high-level, omitting technical details.
Key Moments
Markers derived by PSI from the transcript: the creator did not define chapters.
- Introduction and motivation for length generalization
- Definition of length generalization and introduction to C-RASP
- Empirical correlation between RASP programs and length generalization
- Introduction to planning domains and the need for symbolic generalization
- Formal definition of planning domains and instances
- Examples of planning domains: gripper and lights out
- Motivation for growing alphabet and introduction of C-star RASP
- Main result: C-star RASP guarantees length and symbolic generalization
- Proof sketch: symbolic limit transformer and constraints
- Discussion of limitations and future work
Cited Sources
Concurring Sources
- Huang et al. ICLR paper on length generalization — Referenced in the talk as the formalization of the link between C-RASP and length generalization.
Contribution & Novelties
The talk presents a novel extension of C-RASP to handle growing alphabets, addressing a gap in existing theoretical frameworks. This is significant for planning tasks and for understanding transformer generalization in realistic settings where vocabulary size is large. The introduction of C-star RASP and the symbolic limit transformer provides a new tool for analyzing and designing transformers.
Pour aller plus loin :
- RASP: Thinking Like Transformers — Original paper introducing RASP, foundational to this work.
- C-RASP: Counting Transformers — Paper introducing C-RASP, which C-star RASP extends.
- Length Generalization of Transformers — Related work on length generalization, providing context.
98 words
Radar Profile
The radar profile shows high scores across all dimensions, indicating a technically rigorous and well-sourced presentation. The talk is particularly strong in technical depth and information quality, with slightly lower scores in information quantity due to the focused scope.