The State of Deep Research | AI for Good New York

Value of the Information & Strength of the Argument

The talk provides a valuable overview of the deep research landscape, synthesizing information from multiple sources and offering practical examples. The speaker’s argumentation is coherent, moving from definitions to technical foundations, benchmarks, and limitations. However, the presentation is largely descriptive rather than analytical, with limited critical evaluation of the models or benchmarks. The speaker occasionally interjects personal opinions, but these are not substantiated with data. The talk serves as a good introduction for those unfamiliar with the topic, but it lacks depth for experts.

Scientific Rigor, Source Quality, Title Accuracy

The speaker references several models and benchmarks, but does not provide specific citations or URLs during the talk. The only source mentioned is the speaker’s own book and website (rodeo.ai), which is not a scientific source. The talk’s rigor is moderate: it accurately describes the general concepts but does not delve into technical details or provide evidence for claims. The title accurately reflects the content, which is a state-of-the-field survey. No comments were provided for analysis.

The title accurately reflects the content, which surveys the current state of deep research AI, including models, benchmarks, and future directions.

7/10

The talk provides a broad overview of deep research AI, covering definitions, examples, benchmarks, and limitations. The speaker demonstrates familiarity with the field, referencing multiple models and benchmarks. However, the presentation is largely descriptive and lacks deep technical detail or critical analysis. The information is generally accurate but not exhaustive, and the speaker's personal opinions are present. The talk is not peer-reviewed and is based on the speaker's expertise.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction to the talk and outline.
• 01:08Definition of deep research according to Claude.
• 03:39OpenAI's deep research definition and capabilities.
• 05:16Gemini Deep Research overview.
• 05:39Example from Perplexity Deep Research on quantum computing and cybersecurity.
• 09:01Discussion of DeepSeek R1 and its open-source nature.
• 10:23Explanation of reinforcement learning in the context of reasoning models.
• 11:17Introduction to AutoGen framework from Microsoft.
• 12:36MetaGPT and its application to software engineering.
• 14:08Search-R1: combining reasoning with search via reinforcement learning.
• 16:16ReasonRAG: rewarding process and outcome in reasoning.
• 17:21Multi-agent systems: ManuSearch example.
• 19:20DeepResearcher: another multi-agent deep research model.
• 20:20EvolveSearch: combining supervised fine-tuning and reinforcement learning.
• 22:56Benchmark GPQA: graduate-level science questions.
• 25:36Benchmark GAIA: general AI assistant tasks.
• 27:36Benchmark Humanity's Last Exam: challenging questions to avoid saturation.
• 29:26Benchmark WebArena: interactive web tasks.
• 31:11Benchmark The Agent Company: real-world tasks.
• 32:39Benchmark BrowseComp: web search and browsing.
• 33:34AgentFlow: framework for agent workflows.
• 35:24Flow-GRPO: reinforcement learning for agent workflows.
• 36:14Model Context Protocol (MCP) for tool integration.
• 37:23Agent-to-Agent (A2A) protocol for inter-agent communication.
• 38:41Risks and limitations of deep research.
• 43:30Future directions and conclusion.
• 48:42Mention of the speaker's book 'Foundations of AI Agents Part 1: LLM Agents'.

• Foundations of AI Agents Part 1: LLM Agents (book) — Mentioned at the end of the talk as a resource for further learning.

• OpenAI Deep Research — Official page describing OpenAI's deep research feature, consistent with the talk's description.
• Anthropic Claude Research — Anthropic's research page, relevant to Claude's deep research capabilities.