Strange Geometric Shapes Found Inside AIs — Tom McGrath

Value of the Information & Strength of the Argument

The video provides substantial value by offering an insider perspective on cutting-edge interpretability research, with concrete examples from recent papers and practical demonstrations (e.g., the pirate-style math generalization). McGrath’s arguments are well-structured, moving from the motivation (accelerating science) to specific techniques (SAE-based gradient attribution) and their implications (intentional design, safety concerns). He acknowledges counterarguments, such as the ‘forbidden method’ concerns, and engages with them thoughtfully, strengthening the overall argumentation.

Scientific Rigor, Source Quality, Title Accuracy

The discussion is rigorous, referencing multiple arXiv papers and Goodfire research, all of which are listed in the description. The speaker’s expertise is evident, and he is careful to distinguish between established findings and speculative ideas. The title accurately reflects the content, focusing on the geometric structures found in neural networks. The video includes a brief sponsorship mention, but it does not detract from the scientific content.

The title accurately reflects the content, which focuses on the geometric structures (manifolds, features) discovered inside neural networks through interpretability research.

8/10

The discussion is led by a researcher with deep expertise in interpretability, referencing multiple peer-reviewed papers and providing concrete examples. While the format is conversational and exploratory, the claims are grounded in published research and the speaker's direct experience, lending high credibility.

• 00:00Introduction: Can interpretability speed-run science?
• 02:03The invisible grader
• 06:51What AlphaZero learned from the world
• 12:24Interpretability as a control loop
• 21:54The forbidden method and safer interventions
• 37:36Why models catch hallucinations too late
• 46:19Debug the dataset before training
• 50:44Why neural networks become modular
• 55:57Finding the geometry inside a network
• 1:02:55Why steering falls off the manifold
• 1:12:10A reusable calculator inside Llama
• 1:17:19From abstractions to goals
• 1:25:28Reward hacking, oversight and collusion
• 1:37:23Are sparse autoencoders dead?

• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — Referenced at 00:05:45 in the context of emergent misalignment from reward hacking.
• Acquisition of Chess Knowledge in AlphaZero — Referenced at 00:11:05 as the basis for discussing convergence of learned concepts.
• Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning — Referenced at 00:25:30 in the context of controlled generalization.
• Persona Vectors: Monitoring and Controlling Character Traits in Language Models — Referenced at 00:29:30 in the context of steering model behavior.
• Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability — Referenced at 00:41:14 in the discussion of using interpretability features as rewards.
• Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal — Referenced at 00:47:03 in the context of debugging datasets before training.
• Do Sparse Autoencoders Capture Concept Manifolds? — Referenced at 01:00:26 in the discussion of SAEs and concept manifolds.
• Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior — Referenced at 01:03:04 in the context of steering off-manifold.
• Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts — Referenced at 01:14:20 in the discussion of a reusable calculator inside Llama.
• Measuring Reward-Seeking via Contrastive Belief Updates — Referenced at 01:29:35 in the context of reward hacking and oversight.
• Intentional Design — Referenced at 00:15:44 as the basis for the intentional design concept.
• The World Inside Neural Networks — Referenced at 00:56:12 in the discussion of the geometry inside a network.
• A Pragmatic Vision for Interpretability — Referenced at 01:37:28 in the discussion of the future of interpretability.

• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — Supports the claim that models can become misaligned from narrow training, as discussed in the video.
• Acquisition of Chess Knowledge in AlphaZero — Supports the discussion of convergence of learned concepts in AlphaZero.

• Do Sparse Autoencoders Capture Concept Manifolds? — This paper questions whether SAEs can capture the full concept manifolds, aligning with McGrath's own caveats about SAE limitations.

• app.rescript.info