# Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards Dan Bjornn, Lease End | AI Engineer World's Fair 2026 | 16:39 Source: https://www.youtube.com/watch?v=4loPnxvWWhg Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/your-fine-tuned-model-is-tech-debt-a-50x-roi-house-of-cards Published: 2026-08-20 Tags: agent-skills, agents, cost, fine-tuning, prompt-engineering ## TL;DR - Fine-tuning improved a customer-intent classifier, but each repair took about a week and often caused regressions elsewhere. - The model accumulated a calcification tax that locked Lease End into one model provider, one model version, and an aging workflow architecture. - Replacing fine-tuning with skills, prompts, tools, and context reduced fix deployment from about a week to less than an hour, while total cost fell despite higher cost per message. ## Summary Dan Bjornn describes Lease End's LLM messaging application, which classified customer intent for questions, scheduling, reminders, and sales calls. Retrieval worked reasonably well but missed conversational nuance, so the team fine-tuned a classifier across six intent categories. The application eventually generated $12 million in revenue at a 50x ROI, yet production mistakes exposed the cost underneath. Customers who said "Sounds good" or "Good morning" were sometimes called immediately. Each fix required collecting examples, synthesizing more when needed, manually validating and labeling them, fine-tuning, evaluating regressions, and deploying. The loop usually took a week, with training taking about an hour. Bjornn calls the resulting lock-in the calcification tax. The team rebuilt the application around skills, tools, resources, prompts, and context on a model-agnostic framework. Fixes now take less than an hour to deploy as files, accuracy is higher, and total cost is lower, even though each message costs more. ## Key ideas ### Retrieval missed the conversational nuance needed for customer intent [00:01](https://www.youtube.com/watch?v=4loPnxvWWhg&t=1s) Lease End first built a workflow-based application on top of retrieval. It searched a vector database of previously seen messages classified by customer intent. "Call me tomorrow" meant the customer wanted to talk later, while "I've got time now" meant they wanted a call immediately. The approach worked, but Dan Bjornn says it was not good enough because short messages depend on the surrounding conversation. The system needed to distinguish intents such as talking now, scheduling a call, and opting out. That classification controlled the rest of the customer interaction, so errors at this stage affected the whole application. ### Fine-tuning looked reasonable because the task was narrow and high volume [01:54](https://www.youtube.com/watch?v=4loPnxvWWhg&t=114s) Bjornn gives four reasons that made fine-tuning seem like the right choice. Better accuracy mattered because every workflow depended on getting intent right. Smaller fine-tuned models might reduce cost and latency while handling thousands of messages a day. The application also had a narrow supervised task, sorting conversations into six categories. Finally, the team expected its training data to make the system portable: they thought they could pass the data into a new model and get similar results. These were practical reasons, and Bjornn was initially convinced they justified fine-tuning. ### A successful production model could still make embarrassing decisions [03:32](https://www.youtube.com/watch?v=4loPnxvWWhg&t=212s) The fine-tuned classifier supported an application that generated $12 million of revenue at a 50x ROI within a year. It also produced serious mistakes. In the "confused confirmer" example, Lease End confirmed that Tracy's call was scheduled for Thursday at 2:00 p.m. When she replied, "Sounds good," the model answered, "Great, I'm calling you right now." In another production incident, James replied, "Hi, good morning" to an outreach message, and the model immediately called him. Bjornn calls this the "overeager puppy." The revenue proved the system worked at one level, but these failures made its weaknesses clear. ### Repairing a fine-tuned model took a week, with training as the shortest step [06:04](https://www.youtube.com/watch?v=4loPnxvWWhg&t=364s) A repair began by gathering examples of the new failure. If there were too few, the team synthesized more with an LLM. People then validated those examples, sorted them into intent categories, and manually reviewed the labels. Fine-tuning itself usually took about an hour and was the shortest part. The first iteration rarely solved the whole problem. A change would fix the target issue while causing regressions in older behavior, creating a whack-a-mole loop. Gathering data, labeling it, fine-tuning, evaluating, iterating, and deploying usually took about a week. ### The team ranked bugs by how much customer pain it could tolerate [08:39](https://www.youtube.com/watch?v=4loPnxvWWhg&t=519s) Because every retraining cycle was expensive, Lease End triaged issues before starting one. The team asked how often the problem occurred and whether it caused too much customer pain. Some failures needed immediate attention, such as ignoring a customer's repeated preferred call time or failing to return a scheduling payload correctly, which meant the call never got scheduled. The team also looked for a temporary fix that could avoid a retrain. In practice, bugs were ranked by the amount of customer pain the team could tolerate while waiting for a week-long repair process. ### Fine-tuning created a calcification tax through model and architecture lock-in [09:31](https://www.youtube.com/watch?v=4loPnxvWWhg&t=571s) Bjornn calls the accumulated rigidity the "calcification tax." Training data did not transfer cleanly between model versions. Different providers required different data structures, amounts of training data, and training interfaces. Switching models became too costly, so the team stayed with the same model for consistency. The lock-in also affected architecture. Lease End had built its application around workflows in late 2024, when Bjornn viewed them as the standard way to get good production results. As newer architectures appeared, the team could not adopt them because so much effort went into keeping the existing fine-tuned system working. ### Changing skills and context produced better results than changing the model [11:18](https://www.youtube.com/watch?v=4loPnxvWWhg&t=678s) Bjornn's turning point came from using Claude Code for coding tasks. He noticed that the model did not need to change for each task. The team changed the skill, the resources, and the context instead. Better context produced better results. He applied that idea to Lease End's messaging application, although he says it was difficult to admit because he had championed fine-tuning. The team migrated its workflow approach to skills, tools, and resources that the skills could load. This became an early production test of an agentic framework that was already being built. ### The rebuilt system made fixes faster and reduced total cost [12:14](https://www.youtube.com/watch?v=4loPnxvWWhg&t=734s) After the rebuild, a fix involved finding the problem, adjusting a system prompt or affected skill, testing against a curated set of production examples, iterating, and uploading Markdown files to an S3 bucket. The process from discovering a problem to deploying the fix fell from about a week to less than an hour. Cost per message increased because the rebuilt system used better models, but accuracy improved substantially. The framework was model agnostic, so Lease End could use OpenAI, Anthropic, or another model. Total cost still fell because the team spent much less time maintaining the system and retraining it. ## Notable quotes - "But the whole time it was quietly accumulating debt underneath that we didn't see." (03:55) - "This whole process took about a week to gather the data, label everything, go through the fine-tuning process, and iterate, and then deploy." (07:47) - "We ranked our own bugs based on how much customer pain we could tolerate at the moment." (09:05) - "The important part is the context that we're providing to that model." (14:30) - "So, finally, fine-tune only when you literally cannot call a frontier model." (16:11) ## Tools & references mentioned - Claude Code - OpenAI - Anthropic - Lease End - S3 ## Who should watch - You are considering fine-tuning a classifier because the task is narrow, structured, or high volume, and you want to test the maintenance cost before committing. - Your production model works well on average but fixes take days, regressions keep returning, and the team has started postponing customer-facing bugs. - You are rebuilding an LLM application around prompts, skills, tools, and context and want a concrete comparison with a fine-tuned workflow.