# Trust, but Verify

Shreya Rajpal, Guardrails AI | AI Engineer Summit 2023 | 19:41

Source: https://www.youtube.com/watch?v=9-vGxMoUM9Y
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/trust-but-verify
Published: 2023-11-25
Tags: evals, guardrails, rag, reliability

## TL;DR
- LLM applications are unreliable because model outputs are nondeterministic, and small errors can compound across systems that depend on earlier outputs.
- Guardrails adds a verification layer after every LLM call, with checks for structure, hallucinations, sensitive information, profanity, prompt injection, competitor references, and executable code.
- When a check fails, developers can re-ask the model with the failure details, fix or filter the output, fall back to another system, refuse to answer, or log the failure.

## Summary
Shreya Rajpal argues that LLM applications need a programming pattern built around verification. A database query returns the same answer for the same underlying data, while an LLM can produce different answers to the same question. That makes multi-step applications vulnerable to compounding errors, hallucinations, malformed structure, prompt injection, and unsafe content. Guardrails AI wraps an LLM call in a verification suite whose checks reflect the application. A healthcare chatbot might check for private information and harmful advice, while a code application might run generated code against a sandbox. Failed checks can trigger a re-ask with an explanation, programmatic repair, filtering, fallback, refusal, or logging. Rajpal explains why prompts and fine-tuning do not provide guarantees. She recommends external systems, rules, smaller classifiers, and LLM self-reflection where appropriate. The talk also walks through a retrieval-based chatbot that checks whether each answer is grounded in its source material.

## Key ideas
### LLM applications fail after prototyping because model outputs are nondeterministic
[02:28](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=148s)
Rajpal describes a common experience: an application works during prototyping and becomes unreliable as soon as it ships or another person tests it. A database API gives the correct response for a query about a user's spending over the past month, assuming the underlying data is correct. LLMs do not have that property. Asking the same question several times can produce different responses. When outputs feed later steps in a larger system, errors compound and can grow through the chain.

### LLM use is limited when correctness matters
[04:14](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=254s)
Rajpal says hallucinations are only one problem. Applications also have to deal with incorrect structure and prompt injection. Since English prompts are the main control mechanism, developers cannot reliably enforce requirements through instructions alone. GitHub Copilot is useful when a developer can ignore a wrong suggestion, and ChatGPT can accept feedback through an iterative conversation. That interaction model does not work as well for high-value applications where an incorrect answer cannot simply be discarded.

### A verification suite can decide which model outputs an application may use
[05:58](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=358s)
Rajpal proposes placing a verification suite after every LLM response. The checks depend on the application. A healthcare chatbot might check for personally identifiable or protected health information and profanity. A commercial chatbot might prevent references to competitors. A code-generation system might check that code runs in its target environment. A text-generation system might test whether a summary is true and grounded in a trusted source. The application uses an output only when its checks pass.

### Guardrails turns failed checks into actions such as re-asking or repair
[08:51](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=531s)
Guardrails AI wraps an LLM callable with a guard that validates both inputs and outputs. When a check fails, the system identifies the violated constraint and applies a policy. It can re-ask the model with the generated text and the rule violation, filter the response, try to fix it programmatically, fall back to another system, refrain from answering, or log the failure without taking an active action. Re-asking is useful when the application can afford the extra latency, tokens, and cost.

### The framework combines declarative specifications, validators, and prompt compilation
[09:06](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=546s)
Rajpal explains that developers can create guards from a declarative model specification such as XML or RAIL, from models that implement particular validation criteria and structures, or from string-based implementations. A guard can also include the prompt and the chosen LLM. Guardrails AI provides custom validators, orchestration for validation and verification, a catalog of common validators, and a specification language that compiles requirements into prompts for the model.

### Prompts and fine-tuned models do not guarantee that requirements will be followed
[11:56](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=716s)
Rajpal rejects the idea that prompt engineering or retrieval-augmented generation can guarantee correct outputs. She describes an experiment that ran the same LLM evaluator benchmark five times with no parameter changes and zero temperature, yet produced different numbers. Fine-tuning also takes datasets, training across hyperparameters, and model serving. Commercial APIs add another problem because providers can update model versions without giving developers control. A prompt that worked previously can stop working after an update.

### External systems and small classifiers provide stronger checks for many constraints
[13:57](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=837s)
Rajpal says the right guardrail depends on the problem. For text-to-SQL, connecting generated queries to a sandbox containing the database and schema improved correctness. Rule-based heuristics can enforce simple facts, such as checking that an extracted interest rate has a percentage sign. Traditional machine learning or high-precision classifiers can detect toxicity or harmful advice without using a full LLM. She compares this choice to using a smaller tool for a smaller job, since these checks can be more reliable and deterministic.

### Grounding checks test whether a response can be traced to its source
[16:01](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=961s)
For a chatbot answering questions over help-center articles, Rajpal defines correctness as avoiding hallucinations, profanity, and competitor references. The grounding guardrail checks whether every LLM utterance has support in a source of truth. The implementation can use embedding similarity, natural language inference classifiers, and LLM self-reflection. In her example, the model invents where a password setting exists. The verification step identifies the hallucination, builds a re-ask prompt with the context, and returns a corrected answer after verification passes.

## Notable quotes
- "Machine learning is fundamentally non-deterministic." (02:28)
- "Only use large language model outputs if your verification suite passes." (08:31)
- "If possible, some of the guardrails should use smaller classifiers, they're much more reliable and deterministic." (15:18)
- "You want to make sure that every output that is generated, you're able to pinpoint to where in the context your response came from." (16:47)

## Tools & references mentioned
- Guardrails AI
- ChatGPT
- AutoGPT
- GitHub Copilot
- retrieval-augmented generation
- RAIL
- Alex Gravel
- Predibase
- Apple
- Drive.ai

## Who should watch
- You are shipping an LLM feature that looked good in a demo but produces inconsistent answers in testing or production.
- Your application needs checks for grounded answers, private information, unsafe advice, prompt injection, competitor references, or executable generated code.
- You want practical options for handling failed LLM outputs without relying only on prompts or retraining a model.

## Related talks

- [LLM Safeguards: Security, Privacy, Compliance, and Anti-Hallucination](https://aietalks.com/talks/llm-safeguards-security-privacy-compliance-and-anti-hallucination) (Daniel Whitenack, Prediction Guard, 34:10)
- [Navigating Challenges and Technical Debt in LLMs Deployment](https://aietalks.com/talks/navigating-challenges-and-technical-debt-in-llms-deployment) (Ahmed Menshawy, Mastercard, 16:15)
- [Building Blocks for LLM Systems & Products](https://aietalks.com/talks/building-blocks-for-llm-systems-products) (Eugene Yan, Amazon, 17:24)
- [Lessons from the Trenches: Building LLM Evals That Work IRL](https://aietalks.com/talks/lessons-from-the-trenches-building-llm-evals-that-work-irl) (Aparna Dhinakaran, Arize AI, 18:49)
- [Architecting and Testing Controllable Agents](https://aietalks.com/talks/architecting-and-testing-controllable-agents) (Lance Martin, LangChain, 2:21:54)