# Building Blocks for LLM Systems & Products

Eugene Yan, Amazon | AI Engineer Summit 2023 | 17:24

Source: https://www.youtube.com/watch?v=LzeC1AQ-U5o
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-blocks-for-llm-systems-products
Published: 2023-11-02
Tags: evals, guardrails, human-in-the-loop, rag

## TL;DR
- Evals should be built for the specific task, starting with a small annotated set and becoming automated so teams can test changes quickly.
- Retrieval-augmented generation adds useful external knowledge, but document ranking still matters and language models can give confident answers when retrieved context is irrelevant.
- User feedback becomes valuable training and evaluation data when product interactions make acceptance, rejection, and revision choices visible.

## Summary
Eugene Yan presents practical patterns for putting language models into production systems. He focuses on evaluations, retrieval-augmented generation, guardrails, and collecting feedback. Evals are the foundation because they show whether prompt changes, retrieval, or fine-tuning actually improve a system. Yan recommends task-specific tests, small initial datasets, deterministic checks where possible, and human review for open-ended outputs. Retrieval adds current knowledge, but models still struggle when the relevant document is buried among distractors or when all retrieved context is irrelevant. Guardrails can use moderation, natural language inference, sampling, or a stronger model to detect unsafe or unsupported output. Feedback then feeds future evals and fine-tuning datasets. Examples from GitHub Copilot and Midjourney show how ordinary product actions can provide useful signals without asking users to complete separate surveys.

## Key ideas
### Evals are the foundation for changing an LLM system safely
[00:54](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=54s)
Yan treats evaluations as both development guides and test cases. They help determine whether prompt engineering, retrieval augmentation, or fine-tuning has changed performance at all. Teams can run them before deploying changes, then repeat them after each experiment. Academic benchmarks are difficult to compare because formatting and prompt wording can alter results, and benchmark performance may not match the application. Yan recommends starting with a small, task-specific set. One example uses 40 domain questions. Automated tests can be simple when the output has a known answer, such as checking SQL execution or matching extracted JSON keys and values.

### Open-ended outputs need both automated checks and human review
[05:15](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=315s)
Different tasks support different evaluation methods. Content moderation can use precision and recall for toxicity, bias, and hallucination detection. SQL and JSON extraction can be checked against expected results. Dialogue and other open-ended tasks may require a strong language model to judge outputs, but that can be expensive. Yan cites an example where 60 GPT-4 evaluations cost a lot. He also says that eyeballing still matters. For some outputs, people found it more useful to inspect examples directly than to rely only on automated scores. Human review is a final check, although it cannot scale to every prompt update.

### Retrieval quality still matters when context windows get larger
[05:51](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=351s)
Retrieval-augmented generation lets a model use external knowledge as input context instead of relying only on its stored knowledge. Yan describes an experiment with 20 short documents, where one contains the answer and the others are distractors. Question-answering accuracy is highest when the answer appears in the first retrieved document. It is lower when the answer appears in the middle, and it is better again when the answer is last. Even with perfect retrieval and the answer in the top position, accuracy reaches only 75 percent in the example. Larger context windows therefore do not remove the need to rank relevant documents highly.

### Models may answer irrelevant questions instead of declining
[07:54](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=474s)
Yan gives a recommendation example using a list of science-fiction films and a question about whether the user would like Twilight. The model recognizes that Twilight is a different genre, but still recommends E.T. because of its interspecies relationships. This happens because models are tuned to be helpful and tend to produce an answer even when the available context is a poor fit. Yan suggests borrowing from information retrieval, search, and recommendation systems. A relevance threshold can exclude documents that are too far from the query before they reach the generation step.

### Sentence-level factuality checks catch errors that document-level checks miss
[09:51](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=591s)
Guardrails can check safety through tools such as the OpenAI moderation API, but Yan spends particular attention on factual consistency. He applies natural language inference by treating the source document as the premise and each summary claim as a hypothesis. The claim can be classified as entailment, neutral, or contradiction. Running this check at sentence level is more useful than checking the whole summary. If one sentence is wrong, a document-level score may still call the summary correct, while sentence-level evaluation can identify the specific error. He also mentions generating several summaries and comparing them, or asking a strong language model for a score, though simpler methods can cost less and work better.

### User feedback is a transferable asset for future evals and fine-tuning
[12:28](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=748s)
Feedback shows what customers like and dislike, then becomes data for evaluations and fine-tuning. Explicit feedback, such as thumbs-up and thumbs-down buttons, is often sparse. Implicit feedback is easier to collect but noisy. Copying a code snippet does not prove that the code is correct, so treating every copied snippet as positive training data would add bad examples. Products need interaction patterns that give feedback a clearer meaning. Yan points to GitHub Copilot, where accepting, rejecting, or requesting another suggestion happens repeatedly, and Midjourney, where rerunning, varying, or upscaling an image provides different signals about user preference.

### Existing retrieval and product systems can reduce implementation effort
[16:16](https://www.youtube.com/watch?v=LzeC1AQ-U5o&t=976s)
Yan advises teams to reuse techniques that already work in information retrieval and recommendation systems. BM25, metadata fetching and matching, two-stage retrieval, ranking, and filtering can all help put relevant items first. These systems have been optimized for selecting and ordering useful results, so an LLM application does not need to rebuild every component from scratch. He also says that user experience affects how effectively people can use a model and provide feedback. GitHub Copilot and ChatGPT make models available inside an editor or chat window instead of requiring users to call an API directly.

## Notable quotes
- "Evals help us understand if our prompt engineering, our retrieval augmentation or our fine tuning is doing anything at all." (00:54)
- "We should build evals for a specific task and it's okay to start small." (03:58)
- "Even if the answer is in the context and in the top position, accuracy is only 75%." (07:30)
- "Your evals and fine tuning data set, that's your transferable asset that you can always use." (12:49)
- "Eyeballing doesn't scale. It's good as a final vibe check but it just doesn't scale." (16:03)

## Tools & references mentioned
- Andrej Karpathy
- OpenAI
- OpenAI Moderation API
- MMLU
- Anthropic
- GitHub Copilot
- Midjourney
- ChatGPT
- BM25
- information retrieval
- recommendation systems

## Who should watch
- You are building an LLM feature and need a practical way to test prompt, retrieval, or fine-tuning changes before deployment.
- Your retrieval pipeline returns plausible context, but the model still answers incorrectly or refuses to say that the context is irrelevant.
- You want product interactions such as accepting code or revising an image to produce useful evaluation and fine-tuning data.

## Related talks

- [Lessons from the Trenches: Building LLM Evals That Work IRL](https://aietalks.com/talks/lessons-from-the-trenches-building-llm-evals-that-work-irl) (Aparna Dhinakaran, Arize AI, 18:49)
- [How to Construct Domain Specific LLM Evaluation Systems](https://aietalks.com/talks/how-to-construct-domain-specific-llm-evaluation-systems) (Hamel Husain, Independent consultant & Emil Sedgh, Rechat, 18:45)
- [Best Practices for Evaluating Large Language Model Applications with llmeval](https://aietalks.com/talks/best-practices-for-evaluating-large-language-model-applications-with-llmeval) (Niklas Nielsen, Log10, 09:33)
- [Judging LLMs](https://aietalks.com/talks/judging-llms) (Alex Volkov, Weights & Biases, 18:39)
- [Build Evals That Actually Matter](https://aietalks.com/talks/build-evals-that-actually-matter) (Nick Ung & Akshay Sharma, Lyft, 37:45)