# Do Not Let Models Invent Numbers

Vinoo Ganesh, Kepler | AI Engineer NYC 2026 | 20:58

Source: https://www.youtube.com/watch?v=dZgA6btnb_s
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/do-not-let-models-invent-numbers
Published: 2026-10-25
Tags: guardrails, rag, search, structured-outputs

## TL;DR
- Kepler makes retrieval find cited facts, deterministic code perform calculations, and a post-processing filter remove figures that cannot be verified.
- Financial figures need context such as unit, scale, period, and source because the same number can mean different things in a filing.
- Plain RAG produced thousands of refused or wrong citations in Kepler's evaluations, so semantic retrieval alone is not enough for auditable financial research.

## Summary
Vinoo Ganesh explains why financial software cannot let language models freely produce figures. Kepler separates the work into retrieval, deterministic computation, and verification. Its citation registry extracts facts from more than 30 million SEC documents and records their source, value, unit, scale, period, and other metadata. The model plans which facts and tools to use, while software retrieves data, performs arithmetic, builds tables, and checks citations. Kepler also applies hybrid search, re-ranking, document-level authorization, and tenant isolation across public and internal data. If a number cannot be matched to a trusted source, the system removes it. Ganesh is direct about the tradeoff: this makes Kepler less useful for creative speculation, but it prevents the platform from producing unverifiable financial figures. In Kepler's evaluations, native RAG refused 10,981 citations and included about 3,810 wrong numbers. The guarantees cover traceability and computation, not every part of financial analysis or the replacement of analysts.

## Key ideas
### Financial analysts need every displayed number to be traceable
[01:48](https://www.youtube.com/watch?v=dZgA6btnb_s&t=108s)
Ganesh starts from a finance rule: an analyst should never show a client or portfolio manager a number that cannot be explicitly validated. A statement such as revenue growing 12.4% to 418.2 billion contains several claims that must be correct. Attaching a source after a model has answered does not solve the problem. Retrieval may select the wrong document, or it may select the right document while confusing a value with a page number, a currency, a scale, or a reporting period. Models also do not reliably perform financial arithmetic. Kepler therefore separates sourcing and computation from language generation.

### Kepler lets models cite facts while deterministic code performs calculations
[03:57](https://www.youtube.com/watch?v=dZgA6btnb_s&t=237s)
Kepler's architecture has three parts. Retrieval finds the right source, deterministic software performs the math, and a post-processing filter checks the output. The model can plan which tools and retrieval steps to use, but it does not write the financial calculation itself. Ganesh says this applies to growth rates, margins, multiples, and other derived figures. When a number is not supported by a verifiable citation, the platform is structurally prevented from returning it. This design treats the model as a planner and the software layer as the place where facts and arithmetic are controlled.

### The citation registry turns filing data into typed, clickable facts
[04:30](https://www.youtube.com/watch?v=dZgA6btnb_s&t=270s)
Kepler processed more than 30 million SEC documents and registered numerical facts from filings. These include revenue, income statement and balance sheet values, and figures stated in management discussion and analysis. Ganesh describes roughly 20,000 citable anchors and about 21,000 citations correctly extracted by the filter. Each fact can point back to a source document. The registry also records context that a raw number lacks. A revenue figure in a 2025 10-K may describe 2023, while a table's scale may indicate billions even when the cell only shows 1. The system annotates those relationships before producing an answer.

### A question becomes a plan over facts, tools, and source updates
[06:03](https://www.youtube.com/watch?v=dZgA6btnb_s&t=363s)
For a question about adjusted EBITDA, Kepler may need facts from several 10-Ks, different years, an investor presentation, and an earnings call. It also needs to account for newly disclosed information or a restatement. The language model plans this work with knowledge of the available tools, while deterministic components retrieve facts, calculate values, build tables, and register the resulting citations. Ganesh's example shows why a citation is more than a document link. The system must preserve the meaning of each value and connect the final answer to the exact inputs and operations that produced it.

### Search must combine structured facts with retrieval across internal documents
[08:27](https://www.youtube.com/watch?v=dZgA6btnb_s&t=507s)
Kepler gives each financial fact a value, unit, scale, and period. It cannot rely only on XBRL because Ganesh says XBRL can be incorrect and finance teams also use private material such as investment memos, OneDrive files, SharePoint, Box, and Outlook. The architecture structures data where possible and applies search where it is not structured. Its Turbopuffer namespace combines BM25 and hybrid search. Kepler searches widely, uses approximate nearest neighbors, applies reciprocal re-ranking, then uses a cross encoder and source-aware re-ranking to return material from sources it trusts more.

### Tenant isolation checks authorization at the document level
[11:30](https://www.youtube.com/watch?v=dZgA6btnb_s&t=690s)
Search quality is not enough when different desks or firms have private research. Ganesh gives the example of preventing one Citadel desk from seeing another desk's documents. Each document hit is reauthorized in Postgres, with checks on the citation, value, and document during the computation. Filters accumulate as the system builds a query, and scoped tokens can only be created by the authorization layer. Signed device information also contributes to the access decision. The goal is to keep internal research within the firm's permitted boundaries while the system combines it with other searchable data.

### Unverifiable figures are removed instead of being presented as useful guesses
[14:42](https://www.youtube.com/watch?v=dZgA6btnb_s&t=882s)
During generation, the model receives references that it must resolve against Kepler's citation registry. If the registry cannot match a figure, the post-processing filter strips that number. Ganesh calls this aggressive because it prevents the model from acting as a speculative thought partner. He says Kepler is therefore unsuitable for work that depends on hallucination or creative invention. The system can sometimes attempt citation repair, where it determines whether an otherwise unsupported statement can be reliably tied to a source. Even then, the central rule remains that a displayed number needs a traceable basis.

### Plain RAG produced thousands of rejected or wrong citations
[16:41](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1001s)
Ganesh contrasts Kepler with connecting Claude, ChatGPT, or another model to a normal RAG tool or vector database. In Kepler's evaluations, the filter refused 10,981 citations, and about 3,810 were straight-up wrong numbers. He argues that semantic retrieval can be useful for some tasks, but it does not provide numerical accuracy and auditability by itself. A model may retrieve a plausible source while selecting the wrong figure, or it may perform a calculation that cannot be checked. Financial systems therefore need retrieval with source controls, deterministic computation, and output filtering.

### The guarantees cover traceability, computation, and isolation, not full financial judgment
[19:46](https://www.youtube.com/watch?v=dZgA6btnb_s&t=1186s)
Kepler guarantees that models cite numbers rather than write them, that arithmetic runs in the compute layer, that tenants are isolated, and that recall is measured. It does not claim to replace a full hedge fund or every analyst task. In the product example, users can click Apple's revenue and growth figures to see the filing, presentation, or other source and inspect the calculations behind the result. Ganesh says the remaining opportunity is to automate the tedious extraction and spreadsheet work while leaving room for human financial judgment. The limits of the architecture are part of its claim.

## Notable quotes
- "An analyst, if you're actually working in finance, you never show a client or your PM a number that they can't explicitly validate or trace." (01:48)
- "The model cites the numbers, but never writes them." (03:57)
- "Retrieval finds the right source, code does the math, and we have a post-processing filter." (03:57)
- "When we don't have a match, we strip that number." (14:53)
- "Models don't do math." (17:47)

## Tools & references mentioned
- Kepler
- Vinoo Ganesh
- Palantir
- Citadel
- Anthropic
- Claude
- ChatGPT
- TurboPuffer
- BM25
- XBRL
- SEC
- Postgres
- Google
- Palantir
- Apple

## Who should watch
- You are building a financial research assistant and need every reported figure to link back to a source.
- Your RAG system retrieves plausible documents but cannot reliably distinguish units, periods, currencies, or derived calculations.
- You need to keep private research separated across teams while searching it alongside public filings.

## Related talks

- [AI Can't Make Up a Number It Can't Look Up](https://aietalks.com/talks/ai-cant-make-up-a-number-it-cant-look-up) (Vinoo Ganesh, Kepler, 18:28)
- [How Kepler Built Verifiable AI for Financial Services](https://aietalks.com/talks/how-kepler-built-verifiable-ai-for-financial-services) (Vinoo Ganesh, Kepler, 22:30)
- [Stop Chatting With Spreadsheets](https://aietalks.com/talks/stop-chatting-with-spreadsheets) (Francesco Galletta & Felippe Felisola Caso, Monzo, 1:50:34)
- [Don't Approve the AI Model. Approve the App](https://aietalks.com/talks/dont-approve-the-ai-model-approve-the-app) (Doneyli De Jesus, Langfuse, 19:20)
- [Why Off-the-Shelf AI Doesn't Understand Money](https://aietalks.com/talks/why-off-the-shelf-ai-doesnt-understand-money) (Udi Menkes, Intuit, 19:50)
