AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]

Item: AI Agents can write 10,000 lines of hacking code in seconds [Dr. Ilia Shumailov]
Rating: 4
Author: PostSingularity Institute

by Machine Learning Street Talk

Applied Sciences & Engineering AI — News & Applications UYComputer science UYQArtificial intelligence (AI)

🎙 Machine Learning Street Talk 👥 218K 📅 October 4, 2025 ⏱ 61 min 👁 15K 📄 expert opinion 🧭 2026-08-15

Available in: English (current) Français

Keywords

AI agentssecurityprompt injectionCAMLsupply chain

Summary

In this interview, Dr. Ilia Shumailov, a former DeepMind AI security researcher, discusses the unique security challenges posed by AI agents. He contrasts safety (average-case performance) with security (worst-case performance) and argues that agents are fundamentally different from human adversaries: they work 24/7, can access all endpoints, and can generate sophisticated hacking tools in seconds. He highlights the vulnerability of large language models to indirect prompt injections, citing his work defending Gemini. He introduces CAML, a system that enforces data flow policies by rewriting user queries into formal programs, preventing sensitive data leakage. He also discusses architectural backdoors, supply chain attacks, and the limitations of current security approaches. The conversation covers the need for fine-grained access control and transparency, and touches on model collapse and academic incentives. Shumailov emphasizes that traditional security measures are inadequate for agentic systems and advocates for a new paradigm.

144 words

Critical Evaluation

Value of the Information & Strength of the Argument

The interview provides valuable insights into the emerging field of AI security, particularly the distinction between safety and security and the unique threat model of AI agents. Shumailov’s arguments are well-supported by his experience and references to concrete research, such as the CAML system and papers on prompt injection. He effectively illustrates the inadequacy of current security measures with vivid examples, such as agents sending unsolicited emails. The discussion is technically deep but accessible, making it valuable for both practitioners and informed laypeople.

Scientific Rigor, Source Quality, Title Accuracy

The interview demonstrates strong scientific rigor, with Shumailov referencing multiple peer-reviewed papers and industry reports, including his own work on CAML and architectural backdoors. The sources are credible and directly relevant to the topics discussed. The title accurately captures the central theme, though it slightly exaggerates the immediacy of the threat. The content is well-structured, and the claims are grounded in research and practical experience.

163 words

Title / Content Match

The title accurately reflects the core theme of the conversation: the security risks posed by AI agents, including their ability to generate hacking code rapidly. It is slightly sensationalized but not misleading.

Quality & Reliability

8/10

The interview features Dr. Ilia Shumailov, a former DeepMind researcher with a strong academic background in security and ML. He provides detailed technical insights and references multiple peer-reviewed papers and industry reports. The discussion is grounded in practical experience and academic research, though it is primarily an opinion-driven interview rather than a systematic review.

Chapters

Cited Sources

Lessons from Defending Gemini Against Indirect Prompt Injections — Discussed in relation to prompt injection vulnerabilities in large language models.
Defeating Prompt Injections by Design (CAML) — Proposed system for enforcing data flow policies in AI agents.
Agentic Misalignment: How LLMs could be insider threats — Referenced in context of insider threats posed by AI agents.
STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES! — Mentioned in relation to model reasoning and behavior.
Machine learning models have a supply chain problem — Discussed in context of supply chain attacks.
Supply-chain attacks in machine learning frameworks — Referenced for supply chain vulnerabilities in ML frameworks.
Architectural backdoors in neural networks — Discussed in relation to architectural backdoors.
Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches — Referenced in discussion of semantic censorship and limitations.
Apache Log4j Vulnerability Guidance — Mentioned as an example of supply chain vulnerabilities.
AlphaEvolve MLST interview — Referenced in context of agent behavior and self-learning.

Concurring Sources

Agentic Misalignment: How LLMs could be insider threats — Supports the claim that AI agents can act as insider threats.
Lessons from Defending Gemini Against Indirect Prompt Injections — Provides evidence of prompt injection vulnerabilities in real-world systems.

Dissenting Sources

STOP ANTHROPOMORPHIZING INTERMEDIATE TOKENS AS REASONING/THINKING TRACES! — This paper cautions against over-interpreting model reasoning, which might contrast with some of Shumailov's claims about agent behavior.

External References

Contribution & Novelties

The interview provides a unique perspective on AI security by emphasizing the fundamental differences between AI agents and human adversaries, and by proposing a novel approach (CAML) to enforce data flow policies. It also highlights the inadequacy of current security measures and the need for new paradigms.

Pour aller plus loin :

Prompt injection attacks — Overview of prompt injection attacks and defenses.
Trusted execution environment — Related concept for secure computation.
Supply chain attack — General concept of supply chain vulnerabilities.

82 words

Radar Profile

The radar profile shows high scores in information quantity, quality, and reliability, with a slightly lower technical level, indicating a technically rich but accessible discussion. The overall high scores reflect the expert's credibility and the depth of the content.

Reliability 8/10

💬 No comments were provided for analysis.