Agentic vs. Vector Search: An Eval-Driven Approach to Coding Agent Performance

Value of the Information & Strength of the Argument

The talk provides a valuable, practical introduction to AI evals, with a concrete example that demonstrates the process. The argumentation is clear and logical, building from basic concepts to a real-world application. The speaker is transparent about the limitations of her eval, which enhances credibility. However, the conclusions are based on a small sample size and a single run, so they should be interpreted with caution. The comparison between agentic and vector search is insightful, highlighting efficiency differences, but the speaker acknowledges that her vector search implementation was basic, which could affect the results.

Scientific Rigor, Source Quality, Title Accuracy

The talk is scientifically rigorous in its approach, with a clear methodology and acknowledgment of limitations. The speaker cites the Cursor article as motivation and uses SWE-bench as a benchmark, but no external sources are provided in the description. The title accurately reflects the content. The talk is based on the speaker’s own work and experience, which adds authenticity but also means it is not peer-reviewed. The small dataset sizes and single-run results limit the generalizability of the findings.

The title accurately reflects the content, which focuses on comparing agentic and vector search for coding agents using an eval-driven approach.

7/10

The talk is based on a real-world eval conducted by the speaker, with clear methodology and transparent limitations. However, the sample sizes are small (10 and 25 test cases), and the results are not peer-reviewed. The speaker openly acknowledges the preliminary nature of the findings.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 01:00Introduction to evals and why they are important
• 04:00Explanation of the four core components of an eval: dataset, task, scoring, and experiments
• 08:00Overview of Braintrust features: Loop, MCP server, CLI, and observability
• 10:00Description of the eval loop: from production logs to dataset to evals to code changes
• 12:00Introduction to the real-world eval: comparing agentic vs vector search
• 15:00Explanation of vector search and agentic search concepts
• 18:00Methodology for building the dataset: using PRs from TypeScript Go and SWE-bench
• 21:00Implementation details: using Claude Code, blocking agentic tools, and fixing trace orphan issues
• 24:00Results: similar accuracy but higher cost for vector search; analysis of traces
• 27:00Limitations and future improvements: multiple trials, better vector search, hybrid approach, larger datasets

• Cursor article on agentic search — Mentioned as motivation for the eval, but no URL provided
• SWE-bench — Used as a source for the second dataset (Django rows)

• SWE-bench — Used as a benchmark for coding agents, consistent with the talk's methodology.