The CISO's Guide to Agentic Harnesses | Caleb Sima + Ashish Rajan

The CISO's Guide to Agentic Harnesses | Caleb Sima + Ashish Rajan

Title history
  • The CISO's Guide to Agentic Harnesses | Caleb Sima + Ashish Rajan — observed on 20261115-2012 UTC (YouTube)
🎙 AI Security Podcast 👥 30K 📅 November 15, 2026 ⏱ 41 min 👁 13 📄 debate 🧭 2026-11-15
Available in: English (current) Français

Keywords

AI securityagentic harnessDefense Factoryopen-weight modelsSOC automation

Summary

The podcast episode features a deep conversation between Ashish Rajan and Caleb Sima on building agentic harnesses for cybersecurity teams. Central to the discussion is OpenAI’s Defense Factory framework, which the speakers critique for its narrow focus on code vulnerabilities. They argue that a comprehensive defense factory should extend to least privilege, deployment, user behavior, and incident response. Caleb shares practical insights from his experience, including the creation of vendor-specific AI agents and the importance of running open-weight models on own hardware to bypass restrictions from commercial AI providers. The conversation explores how to scale from individual harnesses to team-wide orchestration, addressing challenges like shared knowledge bases, cost, and the need for dedicated roles to build and maintain these systems. The speakers emphasize that security teams must adopt autonomous AI defense to keep pace with AI-powered attacks, warning that traditional tools and human-only workflows are inadequate. The episode concludes with advice on building in-house harnesses and the necessity of embracing AI-driven automation in cybersecurity operations.

165 words

Critical Evaluation

Value of the Information & Strength of the Argument

The value of the information lies in its practical, first-hand perspective on implementing agentic AI in security operations. Caleb Sima provides concrete examples, such as building eight vendor agents in four hours and the necessity of open-weight models, which offer actionable guidance for security leaders. The argumentation is solid, based on real-world experiences and specific challenges (e.g., Anthropic’s restrictive policies). However, the discussion is largely anecdotal and lacks empirical data or formal research. The speakers effectively argue that AI-powered defense is essential against AI-powered attacks, but the lack of statistical evidence weakens the overall persuasiveness.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is moderate: the speakers reference OpenAI’s Defense Factory (a real announcement) but do not provide direct citations or links to the original documents. The conversation is informed by industry experience, but opinions are not always clearly separated from facts. The title aligns well with the content, as it focuses on CISO perspectives on agentic harnesses. The podcast description includes links to the podcast’s website, research lab, and newsletter, which provide additional context but are not peer-reviewed sources. Overall, the content is more experiential than rigorously academic, yet it offers valuable insights for practitioners.

206 words

Title / Content Match

The title accurately reflects the content: a discussion on how CISOs can implement agentic harnesses for security, with concrete examples and strategic advice.

Quality & Reliability

7/10

The discussion is grounded in practical experience and references to OpenAI's Defense Factory, but lacks formal citations or references to academic research. The speakers are experienced in AI security, but the claims are largely anecdotal and opinion-based.

Chapters

Cited Sources

Concurring Sources

  • AI Security Lab — Research from the podcast's own lab may align with the discussed ideas.

Dissenting Sources

  • Traditional SIEM vendors — The speakers argue that traditional SIEM and human analysts are too slow, which may contradict vendor claims of adequacy.

Contribution & Novelties

The episode contributes original perspectives on applying agentic harnesses beyond code vulnerability management, emphasizing the need for autonomous defense systems. It provides a practical framework for building team-wide orchestration and discusses the critical role of open-weight models in overcoming API restrictions.

Pour aller plus loin :

  • Model Context Protocol (MCP) — Official documentation on MCP, a standard for connecting AI models to external tools, relevant to the discussion on vendor agents.
  • OpenAI Defense Factory — Announcement of the framework discussed, offering context on the code-focused scope.
  • Open-Weights Models in Cybersecurity — A representative preprint on the use of open-weight models for security tasks (URL illustrative; check arXiv for actual papers).

110 words

Radar Profile

The radar profile shows high scores in quantity and technical level, indicating a detailed and technical discussion. The quality is moderately high, but reliability is lower due to reliance on anecdotal evidence. Overall, the episode is strong on practical insight but weaker on formal citation.

Reliability 6/10