# Your LLM Ran Out of Knowledge: Now What?

 | AI Engineer Summit 2025 | 12:51

Source: https://www.youtube.com/watch?v=ya_9_niq2as
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-llm-ran-out-of-knowledge-now-what
Published: 2025-02-22
Tags: human-in-the-loop, prompt-engineering, structured-outputs

## TL;DR
- LLMs perform well in fields such as coding and mathematics because those fields have large amounts of structured training material.
- Domain experts can supply explicit rules and heuristics when important knowledge is informal, experience-based, or missing from written records.
- A pipeline can classify and standardize a problem, apply domain rules, and send the result to a reasoning model, while a subject-matter expert reviews the scenarios.

## Summary
The talk presents a way to use reasoning models in domains with little suitable training data. The speaker contrasts coding and mathematics, where large collections of structured material exist, with corporate negotiations and geopolitics, where much of the useful knowledge is informal and remains in practitioners' heads. The proposed approach gives a model domain-specific heuristics and explicit rules. A parsing stage identifies the type of problem and reformats the input into a standard structure. The system then selects the relevant rule set and passes the structured scenario to a reasoning model, with WorldSim adding real-world constraints in the geopolitics example. The demonstration shows the model applying rules such as leverage and control thresholds and removing scenarios that breach them. The speaker says the approach is still being developed, and insists that a subject-matter expert remain in the loop. Its practical benefit is faster exploration of many scenarios for human review.

## Key ideas
### Some valuable domains lack the written material needed to train a model
[00:01](https://www.youtube.com/watch?v=ya_9_niq2as&t=1s)
The speaker says models do well at coding, mathematics, law, and physics because those fields have large bodies of structured information that can be converted into training data. Corporate negotiations are different. Public reporting may reveal that one company wants to acquire another and may later report the outcome, but it usually does not describe what happened in the negotiation room. MBA courses may provide guidelines, yet the specific discussions and decisions remain unknown. This creates a gap in fields where practical wisdom is held by people and never recorded. The speaker argues that risk and security practitioners share responsibility for this gap because they often do not write their methods down.

### Rules of thumb can give a model practical guidance that training data does not provide
[03:16](https://www.youtube.com/watch?v=ya_9_niq2as&t=196s)
The proposed solution copies a familiar workplace pattern. A recent graduate may be intelligent but lack experience, so a manager gives that person guidelines, rules of thumb, or heuristics for a particular domain. The person can then apply those rules while developing personal knowledge. The system uses the same idea with an LLM. It supplies domain-specific rules and asks the model to apply them on top of its reasoning ability. This gives the model guidance about how to work in a field where the relevant wisdom is absent from its training material. The speaker presents this as a way to help under-served professions use capable models.

### The system first classifies and standardizes each problem
[04:31](https://www.youtube.com/watch?v=ya_9_niq2as&t=271s)
In the demonstration, the user submits a long prompt about an intelligence estimate. The system first determines what kind of problem it has and which rules should apply. It also reformats the question into a standard structure. The speaker says this standardization helps models work more effectively because every scenario reaches the reasoning stage in a consistent format. The front end allows a user to edit the parsed scenario before it continues. This preparation happens before the main simulation or planning model receives the request. The result is a pipeline that turns an open-ended user input into a structured scenario with a known problem type.

### Prompt-based matching replaces a tool-use approach that did not work well enough
[05:05](https://www.youtube.com/watch?v=ya_9_niq2as&t=305s)
Behind the front end, a dedicated LLM prompt matches the user input to one of the available heuristics. The speaker calls this heuristic match broadly similar to tool use, but says tool use did not work as well as hoped in this case. The matching stage identifies the rules that should apply to the problem. The system then parses the scenario into the required structured format. This separation makes the later simulation more consistent because the simulator receives the information in a predictable shape. The speaker also keeps the rules in separate sets, which makes them easier to edit and prevents the overall prompt from becoming too unwieldy.

### Clear binary rules can produce a sophisticated domain model when combined
[06:24](https://www.youtube.com/watch?v=ya_9_niq2as&t=384s)
The demo contains separate scenario rules for negotiations and geopolitics. The rules use direct instructions such as what the model must or must not do. They are intended to be provable rather than vague. One example says that prioritized agreements creating the highest combined value are provable. Another refers to having three or more independent providers for critical resources. Individually, these rules are simple. Combined with a higher-level heuristics prompt, they create a more detailed model of corporate negotiation or geopolitical strategy. The speaker compares them to the rules that might be given to a consultant. The separate rule files also let practitioners adjust guidance for different situations.

### WorldSim adds physical and institutional constraints to geopolitical reasoning
[08:53](https://www.youtube.com/watch?v=ya_9_niq2as&t=533s)
After the system identifies and reformats the problem, it sends the scenario to a reasoning model. In the geopolitical example, the speaker uses Anthropic together with a WorldSim version. WorldSim provides a more realistic view of the world and adds constraints that the model should consider. These include the physical geography of the world and institutions such as the UN and WTO. The speaker says the approach is not tied to one model at this stage. The model can be changed according to what the user prefers or expects to perform best. The external simulation is especially useful when the problem depends on real-world conditions beyond the supplied heuristics.

### The output shows whether the model actually applied the supplied rules
[09:50](https://www.youtube.com/watch?v=ya_9_niq2as&t=590s)
The results are returned in the format defined by the system, so the user can inspect both the structure and the reasoning. The output refers to rules such as two times leverage and greater than 50 percent control. In one example, scenario 5 breaches a rule and is eliminated. The speaker says the point is not to prove that this specific geopolitical approach is the best one. The demonstration shows that the model is using its reasoning ability while also applying the domain rules it was given. This makes the model's behavior easier to inspect against the intended heuristics.

### Human experts should review the scenarios even when the system generates them quickly
[10:57](https://www.youtube.com/watch?v=ya_9_niq2as&t=657s)
The speaker describes the approach as still in progress, although the first results are positive. A subject-matter expert supplies tight rules and heuristics, while the model applies them alongside its reasoning tools and WorldSim. The system can work through many options and produce an optimum solution or multiple scenarios. The speaker still wants a human expert in the discussion. The speed advantage is that a machine can generate dozens of scenarios in a fraction of the time a person would need. Experts can then challenge assumptions, examine harder problems, consider more options within a limited time, or inspect approaches they might not have considered.

## Notable quotes
- "The problem that we have is that we also have other areas where we don't have well structured information." (01:00)
- "We're going to give them a set of rules to follow for specific domains and ask them to apply those on top of the very powerful reasoning capabilities that they now have." (03:52)
- "We want to give it quite clear rules, but by the time you combine these together you quite a sophisticated model." (07:05)
- "I would always want a human in the loop." (11:25)

## Tools & references mentioned
- OracReasoningEngine
- Anthropic
- WorldSim
- Wall Street Journal
- CNBC
- Financial Times
- UN
- WTO

## Who should watch
- You work in a field where experienced practitioners know how decisions are made, but that knowledge is rarely written down for model training.
- You are building an LLM workflow for negotiations, geopolitics, risk, security, or another specialized domain with limited examples.
- You need to generate many structured scenarios for an expert to review, while keeping domain rules visible in the process.

## Related talks

- [Make Your LLM App a Domain Expert: How to Build an Expert System](https://aietalks.com/talks/make-your-llm-app-a-domain-expert-how-to-build-an-expert-system) (Christopher Lovejoy, Anterior, 19:18)
- [Navigating Challenges and Technical Debt in LLMs Deployment](https://aietalks.com/talks/navigating-challenges-and-technical-debt-in-llms-deployment) (Ahmed Menshawy, Mastercard, 16:15)
- [The LLM Triangle: Engineering Principles for Robust AI Applications](https://aietalks.com/talks/the-llm-triangle-engineering-principles-for-robust-ai-applications) (Almog Baku, 26:19)
- [Real ROI: Lessons from Enterprises That Have Already Succeeded with LLMs at Scale](https://aietalks.com/talks/real-roi-lessons-from-enterprises-that-have-already-succeeded-with-llms-at-scale) (Raza Habib, Humanloop, 20:01)
- [10x Development: LLMs for the Working Programmer](https://aietalks.com/talks/10x-development-llms-for-the-working-programmer) (Manuel Odendahl, 1:14:46)