# Productionizing GenAI Models

Lukas Biewald, Weights & Biases | AI Engineer World's Fair 2024 | 22:36

Source: https://www.youtube.com/watch?v=uiq95JYpBGY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/productionizing-genai-models
Published: 2024-10-23
Tags: evals, fine-tuning, observability

## TL;DR
- Generative AI is already in production, but its easy demos create a large gap between a convincing prototype and a dependable application.
- AI engineering is an experimental process, so teams need to passively track prompts, workflows, models, and failures to preserve what they learn and collaborate.
- A serious evaluation framework lets teams compare changes, choose among prompt engineering, fine-tuning, and retrieval-augmented generation, and improve toward a user-facing target.

## Summary
Lukas Biewald argues that generative AI has already entered production, even though many teams still underestimate the work needed after a successful demo. AI applications behave differently from ordinary software because they are experimental and nondeterministic. The learning from failed prompts, workflows, and model changes becomes valuable company knowledge, so teams need passive tracking and reproducibility. He illustrates the process with a voice assistant he built for his daughter. Prompt engineering improved a poor result, model selection helped further, and fine-tuning brought the application to 98% accuracy. Biewald's main advice is to build an evaluation framework before debating techniques such as fine-tuning or RAG. Teams should use small tests for cases that cannot fail, fast evaluations for iteration, and larger scheduled evaluations. They should also start with a lightweight prototype, put it in front of users, collect feedback, and iterate.

## Key ideas
### Generative AI is already in production, but demos hide the hard work
[02:01](https://www.youtube.com/watch?v=uiq95JYpBGY&t=121s)
Biewald says generative AI has moved into production across many companies, including firms that are not especially forward-thinking. More than 70% of the audience reported having LLM applications in production. The problem is that AI is easy to demonstrate and much harder to make dependable. He says CEOs can become overly persuaded by compelling demos and approve systems that are poor in real use. The gap is larger than with ordinary software because AI outputs are harder to predict and evaluate.

### AI engineering is experimental rather than linear
[06:30](https://www.youtube.com/watch?v=uiq95JYpBGY&t=390s)
Biewald contrasts software development with AI development. In ordinary software, teams add code and features, and the product usually moves forward in a fairly linear way. With an LLM application, engineers try prompts, models, and workflows to see what happens. The process is nondeterministic, so a standard CI/CD test will not capture everything that matters. The work requires tools built around experiments rather than only code changes.

### The team's learning is the intellectual property
[07:25](https://www.youtube.com/watch?v=uiq95JYpBGY&t=445s)
For an AI application, Biewald says the model and the final prompt are outcomes. The valuable asset is the learning accumulated while building them, including prompts and workflows that failed. If those experiments are not saved, the knowledge leaves when the engineer who found it leaves. Reproducibility matters because another person must be able to recover the same learning and continue the work. Passive tracking is needed because people will forget to record everything by hand.

### Shorter iteration time helps move an application from demo to production
[08:58](https://www.youtube.com/watch?v=uiq95JYpBGY&t=538s)
Biewald connects experiment tracking to collaboration and delivery speed. When another engineer can pick up previous experiments, the team can avoid repeating failures and test new ideas faster. He describes iteration time as the practical return from these tools. The point is not only to protect company knowledge. It is to make the path from a compelling demo to a usable product shorter.

### A voice assistant improved through repeated experiments
[10:36](https://www.youtube.com/watch?v=uiq95JYpBGY&t=636s)
Biewald built a small Alexa-like assistant after noticing that Alexa did not remember his daughter's favorite song. The system used speech recognition, an on-device language model, and simple skills such as weather, news, music, and arithmetic. The model translated a request such as the weather in Boston into a function call. Latency mattered because speech had to be transcribed and passed through a model, so the system needed a small model running quickly on local hardware.

### Prompting, model changes, and fine-tuning each improved the prototype
[12:19](https://www.youtube.com/watch?v=uiq95JYpBGY&t=739s)
The first default prompt produced 0% accuracy. Prompt engineering raised the result, and switching to a conversation-trained model brought it to 11%. Inspecting errors and incorporating feedback raised it to 75%. Switching from Llama 2 to Mistral produced 79%. Biewald then fine-tuned the model with a manually prepared dataset and additional examples generated by ChatGPT, reaching 98% accuracy. He uses the example to argue that production systems often combine several methods instead of choosing only one.

### Evaluation must replace judging an application by vibes alone
[17:48](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1068s)
Biewald says intuition has some value for catching embarrassing failures, but it cannot tell a team whether a new version is better. A vibes check would not reliably distinguish 75% accuracy from 79% accuracy. Without an evaluation framework, teams cannot decide whether prompt engineering, fine-tuning, retrieval-augmented generation, or another change actually helps their application. Evaluation is the basis for making and comparing improvements.

### Teams need several evaluation speeds and measures tied to user value
[19:01](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1141s)
Biewald describes a layered evaluation practice. Some cases must never fail and should reach 100% performance. Other tests need to run quickly so engineers can assess a change in seconds. Larger evaluation suites can run nightly. The metrics should connect to the application's user experience or the value delivered to customers, although the right measures depend on the application. Production systems can require thousands or tens of thousands of metrics because they have many possible failure modes.

### Small prototypes and early user feedback prevent stalled projects
[19:07](https://www.youtube.com/watch?v=uiq95JYpBGY&t=1147s)
Biewald warns about an enterprise pattern in which teams try to perfect the first stage before putting anything in front of users. The project can remain internal while engineers build the next stage, and never receive real feedback. He recommends a lightweight prototype, early contact with an internal or external user, feedback, and repeated iteration. This is familiar agile product development, but he sees teams forgetting it when they build generative AI applications.

## Notable quotes
- "AI is so easy to demo, so hard to productionize." (03:16)
- "When you're developing AI, it's the learning that's your IP." (07:37)
- "You need to track everything in the background passively to have real reproducibility." (08:42)
- "When you're only testing by vibes, the problem is you can't release a V2." (18:20)
- "You can't improve any of these other things without doing evaluation." (21:33)

## Tools & references mentioned
- Weights & Biases
- CrowdFlower
- Figure Eight
- OpenAI GPT
- Mistral
- Llama 2
- Llama.cpp
- Whisper
- Copilot
- Alexa
- Mycroft
- ChatGPT
- LoRA
- QLoRA
- RAG

## Who should watch
- You have an LLM demo that works in a notebook or scripted path, but you do not yet know how to compare changes reliably.
- Your team is trying prompts, models, fine-tuning, or RAG without a repeatable evaluation set tied to user experience.
- You need several engineers to build on one another's experiments instead of losing context when an experiment fails or its author moves on.

## Related talks

- [The Build-Operate Divide: Bridging Product Vision and AI Operational Reality](https://aietalks.com/talks/the-build-operate-divide-bridging-product-vision-and-ai-operational-reality) (Chris Hernandez, Chime & Jeremy Silva, Freeplay, 12:50)
- [Lessons from building GenAI based applications](https://aietalks.com/talks/lessons-from-building-genai-based-applications) (Juan Peredo, 33:13)
- [7 Habits of Highly Effective Generative AI Evaluations](https://aietalks.com/talks/7-habits-of-highly-effective-generative-ai-evaluations) (Justin Muller, AWS, 25:39)
- [POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments](https://aietalks.com/talks/poc-to-prod-hard-lessons-from-200-enterprise-genai-deployments) (Randall Hunt, Caylent, 19:16)
- [Mastering AI Evaluation: From Playground to Production](https://aietalks.com/talks/mastering-ai-evaluation-from-playground-to-production) (Doug & Carlos Essan, Braintrust, 1:25:08)