When AI Discovers the Next Transformer — Robert Lange

Item: When AI Discovers the Next Transformer — Robert Lange
Rating: 4
Author: PostSingularity Institute

by Robert Lange

Formal & Physical Sciences AI — Models & Research UYQArtificial intelligence (AI)UYQMMachine learning

🎙 Robert Lange 👥 218K 📅 March 13, 2026 ⏱ 78 min 👁 29K 📄 expert opinion 🧭 2026-08-15

Available in: English (current) Français

Keywords

ShinkaEvolveAlphaEvolveco-evolutionquality-diversityAI Scientist

Summary

In this interview, Robert Lange, founding researcher at Sakana AI, discusses the ShinkaEvolve framework, which combines large language models (LLMs) with evolutionary algorithms for open-ended program search. The conversation begins with the limitations of systems like AlphaEvolve, which optimize solutions to fixed problems but require human-provided problems. ShinkaEvolve aims to co-evolve problems and solutions, drawing inspiration from POET, PowerPlay, and MAP-Elites. The architecture uses an archive of programs organized into islands, with LLMs as mutation operators and a UCB bandit to adaptively select among frontier models. Concrete results include state-of-the-art circle packing with fewer evaluations, second place in an AtCoder challenge, and evolved loss functions for mixture-of-experts models. The discussion addresses whether these systems truly think outside the box or are parasitic on starting conditions, with Lange defending the stepping-stone argument. The AI Scientist question is explored, acknowledging that current systems are more co-pilots than autonomous researchers. Lange predicts fundamental transformation of scientific research in 5-20 years, and Tim proposes the thought experiment of alien mathematical artifacts. The episode also touches on the future of work and the potential for AI to discover new architectures.

185 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video offers valuable insights into the intersection of LLMs and evolutionary computation, presenting a novel framework (ShinkaEvolve) with concrete results. The argumentation is strong, as Lange provides technical details and addresses counterarguments, such as the parasitic nature of LLMs. He defends the stepping-stone argument, citing examples from the paper. The discussion is balanced, acknowledging limitations and open questions.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high, with references to multiple arXiv papers and a book. The sources are relevant and credible. The title accurately reflects the content, focusing on AI’s potential to discover new architectures. The discussion is well-structured, with clear explanations of technical concepts. No significant discrepancies between title and content.

125 words

Title / Content Match

The title accurately reflects the central theme: the potential for AI to autonomously discover novel architectures, exemplified by the ShinkaEvolve framework.

Quality & Reliability

8/10

The discussion is grounded in a recent preprint (ShinkaEvolve) and references several peer-reviewed or widely recognized works (POET, MAP-Elites, PowerPlay). The speaker is a founding researcher at Sakana AI, providing expert insight. However, the content is largely conversational and opinion-based, with limited critical examination of limitations.

Chapters

Cited Sources

ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution — The main paper discussed, presenting the ShinkaEvolve framework.
AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery — Referenced as inspiration and comparison for ShinkaEvolve.
Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents — Mentioned as related work on self-improving agents.
Paired Open-Ended Trailblazer (POET) — Referenced as inspiration for co-evolving problems and solutions.
PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem — Referenced as inspiration for auto-curricula.
Automated Capability Discovery via Foundation Model Self-Exploration — Referenced in the context of generating tasks with LLMs.
Illuminating Search Spaces by Mapping Elites (MAP-Elites) — Referenced as a quality-diversity algorithm used in ShinkaEvolve.
Automated Design of Agentic Systems (ADAS) — Referenced in the context of applications.
Discovering Preference Optimization Algorithms with and for Large Language Models (DiscoPOP) — Referenced in the context of applications.
The AI Scientist v2: Automating the Full Research Pipeline — Referenced in the discussion of the AI Scientist.
Why Greatness Cannot Be Planned — Referenced for the stepping-stone argument.
ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering — Referenced in the context of benchmarks.
On the Measure of Intelligence (ARC-AGI) — Referenced in the context of benchmarks.

Concurring Sources

AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery — Both papers use LLM-driven evolutionary search for program discovery, with ShinkaEvolve building on AlphaEvolve's ideas.
Paired Open-Ended Trailblazer (POET) — POET's co-evolution of problems and solutions aligns with ShinkaEvolve's approach.
Illuminating Search Spaces by Mapping Elites (MAP-Elites) — MAP-Elites is a key component of ShinkaEvolve's architecture.

Dissenting Sources

Why Greatness Cannot Be Planned — While the book argues for open-endedness without predefined objectives, ShinkaEvolve still optimizes for a given problem, which may be seen as a tension.

External References

Contribution & Novelties

The video provides an in-depth look at ShinkaEvolve, a novel framework that co-evolves problems and solutions using LLMs and evolutionary algorithms. It highlights the importance of sample efficiency and open-endedness in AI-driven discovery. The discussion offers valuable insights into the challenges and potential of autonomous research systems.

Pour aller plus loin :

Open-endedness: The last grand challenge you’ve never heard of — A comprehensive overview of open-endedness in AI.
Quality-Diversity Optimization: A review of the state-of-the-art — A review of quality-diversity algorithms, relevant to MAP-Elites.
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery — The original AI Scientist paper, providing context for the discussion.

105 words

Radar Profile

The radar profile shows high scores across all dimensions, indicating a well-rounded and informative discussion. The strongest aspects are the quantity and quality of information, while the technical level is slightly lower, reflecting the conversational format.

Reliability 8/10