Aleksandra Bakalova: Discovering Interpretable Algorithms by Decompiling Transformers to RASP

Item: Aleksandra Bakalova: Discovering Interpretable Algorithms by Decompiling Transformers to RASP
Rating: 4
Author: PostSingularity Institute

by Aleksandra Bakalova, Shinting Huang

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQNNeural networks and fuzzy systems

🎙 Aleksandra Bakalova, Shinting Huang 👥 3K 📅 July 2, 2026 ⏱ 39 min 👁 135 📄 original study 🧭 2026-08-16

Available in: English (current) Français

Keywords

RASPdecompilationinterpretabilitycircuit discoverytransformers

Summary

The talk presents a method to automatically extract interpretable algorithms from trained transformers by decompiling them into a new dialect of RASP called D-RASP. The method first reparameterizes a transformer into a D-RASP program that exactly mimics its behavior, then simplifies the program by pruning irrelevant parts and replacing complex matrices with interpretable primitives. The authors demonstrate the approach on two tasks: most frequent token and sorting. For the most frequent token task, the extracted program reveals a uniform attention mechanism that computes a histogram and selects the maximum. For sorting, the program uses a learned attention pattern that attends to the next larger token, enabling sequential output. The method provides direct evidence that transformers often implement interpretable algorithms, and it automates part of the mechanistic interpretability pipeline. The talk also discusses assumptions like linear layer norms and splittable MLPs, and highlights the trade-off between faithfulness and interpretability.

148 words

Critical Evaluation

Value of the Information & Strength of the Argument

The value of the information is high as it introduces a novel method for automatically extracting interpretable algorithms from transformers, addressing a key challenge in mechanistic interpretability. The argumentation is solid: the reparameterization is formally justified, and the simplification steps are empirically validated with match accuracy. The authors acknowledge limitations and assumptions, such as linear layer norms and splittable MLPs, and provide clear examples of extracted programs. The method builds on existing work in circuit discovery and RASP, and the results are convincing for the presented tasks. However, the generalizability to more complex models and tasks remains to be demonstrated.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is good: the method is presented with formal definitions and empirical validation. The sources are primarily the authors’ own work and references to prior work like Weiss et al. on RASP, but no external sources are cited in the talk. The title accurately reflects the content, focusing on decompiling transformers to RASP. The talk is a research presentation, and the methodology appears sound, though it relies on assumptions that may not hold for all transformers. The adequacy between title and content is strong.

201 words

Title / Content Match

The title accurately reflects the content, which focuses on decompiling transformers into RASP-like programs.

Quality & Reliability

8/10

The talk presents a novel methodology with formal reparameterization and empirical validation, but lacks peer-reviewed publication details and external verification.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Cited Sources

Weiss et al. on RASP — Introduced RASP as a symbolic language for transformer computations.
Blog post on D-RASP — Mentioned in the talk for intuitive visualizations.

Concurring Sources

Weiss et al. on RASP — RASP language and its connection to transformers.

Contribution & Novelties

The main novelty is the first method to automatically extract short and interpretable RASP-like programs from trained transformers, combining circuit discovery with program synthesis. This advances mechanistic interpretability by automating the interpretation step. The method is demonstrated on two tasks, showing that transformers often implement interpretable algorithms.

Pour aller plus loin :

RASP: A Language for Neural Network Computation — The original RASP paper by Weiss et al., foundational to this work.
Mechanistic Interpretability — A community resource on interpretability techniques.
Circuit Discovery — A paper on automated circuit discovery, relevant to the pruning step.

95 words

Radar Profile

The radar profile shows high scores in technical level and information quantity, with slightly lower but still strong scores in information quality and reliability. This indicates a technically dense presentation with solid content, though the reliability is tempered by the lack of external verification.

Reliability 7/10

💬 No comments were provided for analysis.