Trust, but Verify: Knowledge Agents for Finance Workflows

Mike Conover, Brightwave21:10 · Apr 2025 · 31K views
Thumbnail for Trust, but Verify: Knowledge Agents for Finance Workflows Watch on YouTube
TL;DR
  1. 1

    Financial research agents need to process large document collections while giving analysts a way to inspect the evidence behind every conclusion.

  2. 2

    Constraining an agent's scope can reduce the chance of degenerate output while better models and end-to-end reinforcement learning remain developing areas.

  3. 3

    The most useful product interface is an interactive research surface where users can inspect citations, findings, timelines, and other details on demand.

Summary

Mike Conover describes Brightwave's research agent for financial diligence, where analysts may need to review thousands of pages, earnings transcripts, filings, or hundreds of vendor contracts under tight deadlines. He argues that these systems need more than chat. They need workflows that break research into document selection, finding extraction, enrichment, error correction, synthesis, and human review. Separate model calls can check whether a finding is supported by its source. Narrower instructions can also produce denser, more focused outputs. The system should expose an audit trail through citations, document passages, structured findings, and interactive follow-up. Conover is honest that complex synthesis, temporal context, and the right model for each step remain unsolved problems. He also argues that users should guide the system with domain judgment, since important information may remain outside the digitized corpus.

Key ideas
00:00

Financial diligence creates impossible workloads for junior analysts

Conover opens with examples from investment work: entering a data room containing thousands of pages, reviewing 80 or 120 companies during earnings season, or examining 80, 800 vendor contracts for early termination clauses. Analysts must find risks quickly and understand patterns across an entire portfolio. He says this work is beyond what people can reasonably do manually under tight deadlines, and that junior analysts are often asked to complete it anyway. Brightwave is aimed at digesting these large financial corpora so professionals can reach conviction faster and spend more time on judgment.

02:09

Knowledge agents change the amount of thought a finance professional can bring to a problem

Conover compares knowledge agents with spreadsheets. Before computational spreadsheets, an accountant might describe the job as running numbers by hand on large sheets of paper. The work was important and demanding, but the tool limited how much analysis a person could perform. Modern systems can digest far more content and accelerate the path from information to useful work. He says the change is not that financial professionals disappear. Their ability to think about a business, market, or portfolio can become more sophisticated because the system handles more of the reading and organization.

03:35

A research agent must make its reasoning over thousands of pages legible

The product problem is how to reveal what a system did with 10,000 pages of material in a form a person can use. Conover says chat is probably insufficient as the final interface. Analysts need visibility into the documents, findings, intermediate beliefs, and connections between facts. The interface should let a user recognize what matters, inspect details when needed, and redirect the system toward an interesting line of inquiry. This makes interaction design part of the agent architecture, because the interface determines how much of the system's work the analyst can verify.

04:16

Local extraction errors compound across chained model calls

Conover describes non-reasoning models as performing greedy local search. If a model extracts organizations from a Reuters article with a 5 or 10 percent error rate, chaining many similar calls can make the overall likelihood of error much worse. He argues that stronger systems will train end-to-end reinforcement learning over tool calls, with API results included in the sequence of decisions. That would let an agent make a locally weaker choice when it improves the final result. He says intelligent tool use with knowledge graphs remains an open research problem.

05:16

Constraining an agent's behavior can reduce degenerate output

While better models, more data, and more compute may eventually dominate, teams still need to build products now. Conover suggests treating the scope of an agent's behavior like a regularization parameter. Limiting what the system is allowed to do can reduce the likelihood that it goes off the rails and produces degenerate output. He compares this with a conversation in which a user repeatedly nudges a model toward a solution. The product should provide that scaffolding directly, because most people with full-time jobs will not spend roughly a thousand hours becoming prompting experts.

07:47

A finance workflow should decompose research into explicit stages

Conover lays out a workflow for investigating a polypropylene resin manufacturer. The agent might find public-market comparables, inspect SEC filings and earnings-call transcripts, use a knowledge graph from earlier deals, select relevant document sets, and distill findings that support or challenge an investment hypothesis. It can then enrich those findings and check them for errors. Intermediate notes are useful because they record what the system believes after its first pass through the data. A separate call can ask whether a fact is entailed by a document or whether an extracted item is actually an organization, which can correct the model's initial answer.

09:58

Synthesis requires weaving facts across documents, and current models still struggle

The difficult part is combining fact patterns from many documents into a coherent analysis. Conover says long inputs are compressed into a much shorter output, and models are better at focused tasks than at producing tens of thousands of coherent words. More specific subthemes can lead to denser and higher-fidelity reports. He also says instruction-tuning data contains relatively few examples of genuinely recombinative reasoning, such as reading many papers and producing a useful synthesis. Temporal details make the problem harder. A merger can change the meaning of financial statements, while contract addenda need evidence and metadata that explain how they relate to other documents.

16:45

The interface should expose receipts and support investigation on demand

Conover presents the report as one view of a richer, high-dimensional structure of findings. Analysts should be able to click a citation and see the source, surrounding context, and the model's reasoning. They should also be able to highlight any passage and ask about its implications, rather than being limited to predefined citations. Structured views can expose items such as a fundraising timeline, litigation, rising capital expenditure, patent disputes, or a factory fire that affected a single-source supplier. This gives the analyst a way to apply domain judgment, pull on an interesting thread, and inspect the evidence behind a conclusion.

"The design problem that you have to solve is how do you reveal the thought process of something that's considering 10,000 pages of content to a human in a way that's useful and legible."03:35
Who should watch
  • You are building an agent that has to read large document collections and need a workflow for selecting sources, checking findings, and synthesizing evidence.
  • Your users work in finance or another high-stakes domain where a polished answer is insufficient without an audit trail and source-level inspection.
  • You are deciding whether chat is enough for an agent product and want concrete interaction patterns for human guidance and details on demand.