# Just-in-Time Ontologies: What Cheap Intelligence Means for Data

Hrishi Olickel, Southbridge AI | AI Engineer NYC 2026 | 21:03

Source: https://www.youtube.com/watch?v=Fa3hQVIKShw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/just-in-time-ontologies-what-cheap-intelligence-means-for-data
Published: 2026-10-12
Tags: context-engineering, data-pipelines, evals, reliability

## TL;DR
- A universal schema fails because source, storage, decision, and model assumptions shape what data means.
- Reliable pipelines separate models that design the work, models that perform it, and models that optimize it.
- Self-optimizing data systems need a stable execution surface, bounded experiments, reusable methods, and a defined failure surface.

## Summary
Hrishi Olickel argues that data transformation becomes more useful when it is built around the decision being made. A universal schema cannot capture what a customer means to billing, support, and compliance at the same time. His alternative is a just-in-time ontology, created when the question is known and stored with its context and evidence. An Ohio campaign finance entity resolution project shows the method. Separate models designed the pipeline, performed individual checks, and searched for better architectures. The team refined definitions, used held-out cases, compared model families, and preserved versioned work segments. Olickel also describes failures that several models accepted, including dropped minus signs that changed net $784,000 into $5.3 million of debt. He says self-optimization requires a stable harness, limits on cost and time, reusable methodology, and a defined way to measure failure. Small pilots and independently checked work segments make later changes easier to inspect and reverse.

## Key ideas
### Data has four schemas, and their meaning becomes clear only when the decision is known
[00:00](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=0s)
Olickel begins with the assumption that transformation is expensive and asks what changes when intelligence becomes cheap. He describes four schemas: the source schema, shaped by what the device or system records; the storage schema, limited by what can be saved; the decision schema, shaped by the question; and the model's prior, which brings assumptions about what should be present. A customer is whoever pays for billing, whoever calls for support, and whoever the organization is responsible for in compliance. The useful structure appears when someone asks a concrete question such as whether to refund an order or extend credit.

### A just-in-time ontology joins recorded context to the decision instead of imposing one universal schema
[04:28](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=268s)
Olickel says he was wrong to promote a single universal schema. The problem is not only that migrations lose organizational support. There is no super schema that can represent every use of the data. Transformation makes information useful by merging the context in which it was recorded with the context of the decision. A just-in-time ontology performs that merge when the decision space is known. The resulting structure can preserve what the answer rests on and how far it was checked, rather than stripping away the conditions that gave the data meaning.

### Data work falls into one-off jobs, one-time migrations, and repeatable pipelines
[05:34](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=334s)
He divides data workloads into three regimes. Write-once, run-once jobs handle a single dataset, information-mining task, or question and then are discarded. Write-many, run-once work contains many smaller and larger workloads built for one migration, with each part checked at its source and against the others. Write-many, run-many pipelines are repeatable systems maintained as new information arrives. Olickel says these systems now cover work at Southbridge such as benchmark design, tracking, coding, and making subtitles. In every regime, the practical requirement is that people can rely on the results.

### Pipeline intelligence should be split between designers, workers, and optimizers
[08:30](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=510s)
Olickel separates three roles that coding agents often place in one context window. Designers break down the problem, write early specifications, and authorize spending. Workers read PDFs, write to databases, and check results inside the pipeline. Designers benefit from slower models that can digest documentation and reason about how a system will behave. Workers need cheaper, faster, specialized models that follow instructions consistently. A third group, which he calls hill climbers or surfers, runs a first version, varies models and budgets, debugs, records what it learns, and searches for a better solution.

### Definitions and held-out cases matter as much as the choice of model
[10:34](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=634s)
In the Ohio campaign finance project, the team tested more than a dozen architectures for deciding whether records belonged to the same person, company, or committee. One model missed all 13 development questions about household links because the question did not define enough evidence. Rewriting the question changed the answers without changing the model or thresholds. As the work scaled, each round fixed a definition, including what counts as a committee, what a filer means, and when two people at one address count as a match. Each of the 23,317 Ohio cases received a label describing its basis and checking.

### Models can silently apply their own assumptions to data
[13:58](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=838s)
Olickel gives several examples of errors that looked internally consistent. A model searched for a field with an underscore while the file used a capital letter and a hyphen, so it counted 15,658 in-kind contributions as ordinary money. Another tried four date formats while the file used a fifth, leaving 99.4% of transaction dates empty even though row counts and dollar totals matched. In a more serious case, models treated negative loan amounts as redundant signs. Net $784,000 became $5.3 million of debt, and multiple models validated the result. Bigger models can hold their assumptions more firmly.

### Different model families can focus review on disagreements, but shared priors still create common errors
[15:41](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=941s)
On 200 held-out families, Jev and Luna agreed on 194 cases, and none of those shared answers was wrong. The six disagreements contained the errors, which made review smaller. Olickel says models from different families tend to have different priors and therefore disagree more often than models from the same group. This approach has a limit. Four model families once produced the same incorrect total because they shared a sign convention that disagreed with the answer key. Teams must identify these recurring assumptions and add specific checks for them.

### Self-optimizing pipelines need a stable surface, bounded resources, reusable methods, and a failure surface
[16:42](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=1002s)
Olickel names four requirements for optimization. A stable deployment surface provides consistent logs, APIs, checkpoints, rollback, and budget enforcement despite changing agent runtimes. Experiments also need limits on cost, time, and access. The system needs reusable know-how from earlier deployments, including debugging techniques and agentic patterns. Finally, it needs a failure surface, such as a manually checked baseline, schema checks, or measured behavior, defined before the work starts. Without a way to tell when a change fails, the system has no dependable basis for improving itself.

### Bounded pilots and reusable checked segments make optimization inspectable
[18:14](https://www.youtube.com/watch?v=Fa3hQVIKShw&t=1094s)
The workflow starts by specifying and splitting the job, then defining how failure will be measured. The team runs a small bounded pilot and uses it to establish a baseline. Working checks move into reusable pieces called codons. These segments can run independently, be validated, and be placed in sequence. Once an architecture works, the team swaps in cheaper or faster models and compares each version with known intermediates. Olickel says the ontology is built when the job needs it, kept with its receipts, and reused when another job needs the same structure. That context allows cost, speed, and accuracy to be traded without losing the evidence.

## Notable quotes
- "There is no concept of a customer that's divorced from the question that you want to answer." (04:28)
- "The whole statewide job came to $7.95 on an $800 budget that was set ahead of time." (12:16)
- "Net of 784,000 became 5.3 million of debt in the end validated by multiple models." (14:44)
- "The fourth and the most important one is a failure surface." (17:47)
- "It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing." (20:06)

## Tools & references mentioned
- Hannes at DuckDB
- DuckDB
- Southbridge AI
- Jev
- Luna
- Fable
- Hank
- Hankweave
- Claude Code
- PI
- Opus
- Astra
- Adam Smith

## Who should watch
- You are building data pipelines that must transform messy records into answers for a specific operational or compliance decision.
- Your team is using agents for data work and needs to separate pipeline design from repetitive execution and system optimization.
- You need a practical way to compare model variants while preserving definitions, checks, budgets, and evidence for later review.

## Editor's note

Hrishi Olickel shows that several model families can accept the same wrong sign convention and produce an incorrect total. Kitaru records every model call and tool result from a real run, then replays the agent against those same inputs and responses. That lets a team test a changed model, prompt, or code against the failure without touching real systems.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- ["Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer](https://aietalks.com/talks/data-readiness-is-a-myth-reliable-ai-with-an-agentic-semantic-layer) (Anushrut Gupta, PromptQL, 17:02)
- [The Rise of CaaS: Context-as-a-Service for Agentic AI](https://aietalks.com/talks/the-rise-of-caas-context-as-a-service-for-agentic-ai) (Omer Primor, Bright Data, 22:20)
- [How Web Data Infrastructure Powers the Next Generation of AI](https://aietalks.com/talks/how-web-data-infrastructure-powers-the-next-generation-of-ai) (Patricija Žemaitytė, Oxylabs, 19:03)
- [Demand-Driven Context: A Methodology for Coherent Knowledge Bases Through Agent Failure](https://aietalks.com/talks/demand-driven-context-a-methodology-for-coherent-knowledge-bases-through-agent) (Raj Navakoti, IKEA, 1:08:15)
- [Designing AI-Intensive Applications](https://aietalks.com/talks/designing-ai-intensive-applications) (swyx, 13:02)