Agentic Financial Reasoning with Knowledge Graphs and LLMs

Value of the Information & Strength of the Argument

The talk provides valuable insights into the importance of structured evidence for LLM reasoning in finance. The argument is well-supported by empirical results from the speaker’s research, including a large-scale benchmark and experiments with multiple models. The speaker effectively demonstrates that evidence organization is a critical factor, often more important than model scaling. The discussion of token efficiency and cost trade-offs is particularly useful. The argumentation is solid, though it would benefit from more discussion of potential limitations and alternative approaches.

Scientific Rigor, Source Quality, Title Accuracy

The talk is scientifically rigorous, referencing the speaker’s own published work and the FinReflectKG dataset. The methodology is clearly described, and the results are presented with quantitative evidence. The title accurately reflects the content. The speaker does not cite external sources beyond his own work, but the research appears to be well-conducted. The talk is based on a single perspective, but the evidence is compelling.

The title accurately reflects the content, focusing on agentic reasoning in finance using knowledge graphs and LLMs.

8/10

The talk is grounded in the speaker's own research, with references to published papers and a large-scale benchmark (FinReflectKG-MultiHop). The methodology is clearly explained, and the results are presented with quantitative evidence. However, the talk is a single perspective and lacks external validation or discussion of limitations in depth.

Markers derived by PSI from the transcript: the creator did not define chapters.

• 00:00Introduction: The key question of building reliable agents for finance.
• 02:00Discussion of financial document complexity and the need for structured evidence.
• 05:00Introduction of knowledge graphs as a grounding layer for agents.
• 10:00Presentation of FinReflectKG and the benchmark creation process.
• 15:00Experimental setup: six evidence protocols and multiple LLMs.
• 20:00Results: KG-linked evidence improves correctness and reduces token usage.
• 25:00Discussion of building agentic systems and evaluation-driven development.
• 30:00Q&A: Elaboration on 'evaluate then develop'.

• FinReflectKG paper and dataset — Mentioned as the paper published around August last year on building and evaluating knowledge graphs at scale.
• FinReflectKG-MultiHop benchmark — Introduced as the large-scale benchmark for multi-hop QA in finance.

• GraphRAG: Unlocking LLM discovery on narrative private data — Supports the use of knowledge graphs for retrieval-augmented generation.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Foundational paper on RAG, relevant to the discussion of retrieval methods.

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Suggests that reasoning LLMs can handle complex tasks without external structure, contrasting with the talk's emphasis on structured evidence.