Demand-Driven Context: What 300 Incidents Reveal About Tribal Knowledge - Raj Navakoti
Keywords
Summary
99 words
Critical Evaluation
Value of the Information & Strength of the Argument
The value lies in presenting a practical methodology to surface tribal knowledge and make it finite. The argument is supported by the experiment’s data: the number of new entities decreases per cycle, showing convergence. However, it’s a single case study and may not generalize. The reasoning is logical and builds on the failure of previous approaches.
Scientific Rigor, Source Quality, Title Accuracy
The talk does not cite external sources, relying on the speaker’s experience and data. The description links to conference pages only. The title matches the content well. No comments provided, so no analysis of public reception.
107 words
Title / Content Match
The title accurately reflects the content, as the talk details a study of 300 incidents to surface tribal knowledge and introduces the Demand-Driven Context methodology.
Quality & Reliability
7/10
The talk presents a case study with specific data (46 entities, convergence after 9 cycles), but it is a single experiment, self-reported, and not peer-reviewed. The methodology is plausible but lacks independent validation.
Key Moments
Markers derived by PSI from the transcript: the creator did not define chapters.
- Introduction and audience question about the most painful task for a software engineer
- First attempt with limited documentation for the SRE agent fails, leading to calling Bob
- Second attempt dumping all Confluence pages via RAG still fails, producing garbage answers
- Introduction of Demand-Driven Context (DDC) methodology: agents ask for missing knowledge
- Experiment with 300 incidents clustered into 20 types, showing convergence to 46 entities after 9 cycles
- Discussion of knowledge saturation and finite knowledge for agents
- Introduction of 'propose knowledge' skill to maintain the knowledge base via MCP and pull requests
Cited Sources
- NDC Conferences — Main conference site mentioned in description
- NDC Oslo — Specific conference site where the talk was recorded
Concurring Sources
Dissenting Sources
Contribution & Novelties
The talk contributes a novel methodology (Demand-Driven Context) that applies TDD principles to knowledge engineering for AI agents. It empirically demonstrates that knowledge required for incident resolution is finite and converges, offering a data-driven approach to curate context. The ‘propose knowledge’ skill also addresses knowledge maintenance.
Pour aller plus loin :
- Test-Driven Development — The inspiration for DDC; TDD emphasizes writing tests that fail before implementing code, similar to letting agents fail to identify knowledge gaps.
- Retrieval-Augmented Generation — A technique used in the talk’s second attempt; combining retrieval with generation, but here it failed when given all documentation.
- Knowledge Graph — A way to structure knowledge; the talk considered building a knowledge graph but found it unnecessary when using a curated entity list.
124 words
Radar Profile
The profile shows high scores in information quantity and moderate scores in reliability, reflecting a data-rich but self-reported case study. The balance suggests a talk that is informative and technically detailed but with limited external validation.