An AI product has to survive skeptical review, where every claim can be checked and challenged.
2
Trust depends on source quality, consistent numbers, visible contradictions, and a clear separation between facts and guesses.
3
Products become usable in serious decisions when every claim has provenance and a named human approves the final output.
Summary
Shawn Chan compares an impressive AI demo with an investment memo. A demo produces a fluent answer that makes people interested for a few minutes. A memo has to survive a room filled with people looking for errors before real money moves. Chan argues that the same qualities that make an AI product trustworthy also make an investment case credible. Sources need different trust levels. Numbers must agree across documents. Contradictions should be shown instead of quietly resolved. Facts and estimates need separate labels. Claims need links to their exact source paragraphs, and a human must approve and sign the final decision. He uses examples involving a wrong AI marketing claim, inconsistent financial figures, invented court cases, and an airline chatbot that made up a policy. His conclusion is practical: finance products will win when a tired reviewer can trust the output without opening seven browser tabs at night.
Investment decisions depend on trust more than polished intelligence
Chan says he has spent 15 years deciding whether his company should spend large sums of money after reviewing papers from confident people. AI-generated papers can have better grammar, cleaner formatting, and a more patient tone than human work, but confidence does not show that a claim is right. After about 200 investment committee meetings and hundreds of pitch decks, his lesson is that money follows trust. He recalls a banker presenting confident but incorrect numbers. When a senior person asked, "Where does this number come from?", the pause told Chan more than years of exams. The source behind each claim matters more than the appearance of certainty.
A demo is judged by surprise, while a memo is judged by survival
Chan separates two kinds of work. A demo takes one clean document and produces one fluent answer, with the aim of making a room react positively for a few minutes. A memo gathers filings, transcripts, notes, and other documents that may disagree. Its job is to survive an argument before real money moves. A phone summarizing an email can sound plausible and still be a demo. An AI system defending why someone deserves a mortgage has to be right and prove its answer. Chan says builders keep confusing these machines, even though the second standard is the one that matters in serious decisions.
A single unchecked sentence can destroy confidence in an entire product
Chan describes a February 2023 marketing demo from a major technology company in which an AI assistant gave a wrong answer about a space telescope. He says the market noticed, the company's stock fell around 8% in a day, and roughly $100 billion in value disappeared. His point is that the demo failed the memo test. Once real money is watching, every sentence can be treated as a claim that needs checking. He connects this incident to the question junior analysts are trained to ask before work leaves the building: "Where does this claim come from? Did everyone check it?"
Chan says an audited filing, an analyst note, an internal email, and a group-chat guess should not receive the same trust. Yet many retrieval systems select the text closest to a question and present it as authoritative. He recalls an expensive AI tool choosing a confident rough guess from a group chat over the actual audited number a few rows away in a filing. The problem was not that the system lacked fluent language. It could not distinguish an accountant speaking under formal scrutiny from a rumor overheard in a group chat. A product used around real money needs to preserve that difference.
Numbers that disagree are evidence of a checking failure
A memo can say revenue growth is 18% on its first page and 17.4% in a table on page 11. Chan says the room is less concerned with the missing 0.6 than with what the inconsistency implies about the rest of the work. He also cites a large American real estate company whose house-buying algorithm was confident about prices that did not match reality. The company wrote off around half a billion dollars, shut the unit down, and dismissed a quarter of its staff. Chan says the model was unsupervised, and the system lacked a basic mechanism to keep its numbers aligned with reality.
Contradictions should be surfaced instead of smoothed away
Chan calls a contradiction a gift because it may contain the most interesting fact in the material. If a CEO gives one gross number on an earnings call and the official filing gives another, the gap deserves investigation. AI systems often do the opposite. They choose the version that reads more smoothly and hide the disagreement. Chan recalls seeing a meaningful difference between a CEO's number and a filing number go unnoticed because nobody had both documents open at once. He says that was luck, and luck is not a control. The product should put the argument in front of a human rather than resolve it invisibly.
Facts and estimates need labels that survive repeated editing
A sentence such as "The company will likely receive approval next quarter" sounds factual, but it is an estimate. Chan says committees need to trust the facts while debating the estimates. If a system merges them into one smooth sentence, reviewers cannot tell which parts are proven. He saw an approval estimate change across three drafts from "approval expected soon" to "approval received." Nobody necessarily lied. The guess gradually acquired the appearance of a fact until nobody remembered its origin. His proposed fix is simple: label estimates with a tag, color, or other marker that remains visible after the content is copied into someone else's slides weeks later.
Provenance and human accountability belong inside the product
Chan's requirements are concrete. Each claim should link directly to its source paragraph and include the source's trust level. Facts and estimates should be visibly separate. The system should block a memo when figures do not match, surface conflicts rather than choose a nicer answer, and record who reviewed what, changed it, and signed it. He rejects a citation tab that forces people to search through references. He also describes an airline chatbot that invented a bereavement discount policy and a tribunal that rejected the airline's attempt to treat the chatbot as a separate legal entity. Chan's conclusion is that software cannot take responsibility. A human has to sign the decision.
"Money doesn't follow intelligence. Money follows trust, and the trust is fragile."04:08
Who should watch
You are building an AI product that will be used in finance, compliance, lending, investment, or another setting where someone must defend the output.
Your system produces polished answers, but reviewers still need to open several documents to check sources, reconcile figures, or find the person responsible.
You are raising money and want to understand how an investor will test the numbers and claims in your pitch deck after the meeting.
