Most teams should begin with a prompted model because it gives them fast iteration and reveals what users actually need.
2
Fine-tuning is useful when cost, latency, or inconsistent output quality becomes a real production problem.
3
The practical fine-tuning workflow is to collect production data, prepare high-quality examples, train, evaluate, and deploy.],
Summary
Kyle Corbitt argues that most teams should not start with fine-tuning. A prompted model lets a team change behavior quickly while learning the real distribution of user inputs. Once the product shape is clear, fine-tuning can produce a smaller, cheaper, faster model with more consistent outputs. Corbitt explains the trade-off: a tuned model specializes in a defined task and loses flexibility outside that domain. He gives three reasons to move from prompting to fine-tuning: production cost, latency, and quality consistency. The workflow is less intimidating than many engineers expect. Teams can log prompts from their existing application, review or relabel examples when needed, train with hosted or open-source tooling, run fast model evaluations and business-specific production tests, then deploy through a hosted provider or self-hosted inference. His advice is practical: use prompting to discover the task, then tune when the economics or reliability justify the extra work.
A prompted model is usually the right starting point
Corbitt says that most people in the audience probably do not need fine-tuning yet. A prompted model is easier to change, so it supports faster product iteration while a team is still learning what the application should do. It also exposes the actual input distribution from users. By logging prompts in production, a team can see which tasks users send, including unexpected uses. That information becomes the basis for deciding whether fine-tuning fits and for building the training set later. His recommendation is to start with a capable prompted model when the problem allows it, then move up the curve after the product shape is clear.
Fine-tuning can move the cost and quality frontier
Corbitt frames model choice as a quality-versus-cost frontier. Larger and more heavily trained models usually produce better zero-shot results, but they cost more. Prompting techniques that use extra inference compute, such as a mixture-of-agents approach, can push quality higher, although they add cost and latency. Fine-tuning can take the high-quality behavior produced by those methods and put it into a smaller model. In Corbitt's internal evaluations on four real customer datasets, fine-tuned versions of Llama 3 70B and another Llama 3 model performed substantially better than their prompted versions and above the prompted base models. The claimed benefit is better quality at the same inference budget.
Fine-tuning has an upfront cost because someone must collect data and verify that the model works. It also narrows the model's scope. Corbitt says tuning makes a model very good at one specific task, while it becomes less useful for unrelated inputs. That makes task definition important before training. The application should send the tuned model inputs that resemble its training data. Out-of-domain requests will not receive the same performance improvement. This specialization is the reason fine-tuning should not happen automatically at the start of a project. Teams need to understand what the production task actually contains before optimizing for it.
Cost, latency, and consistency are the practical triggers
Corbitt gives three reasons to move from a prompted model to a tuned model. The first is unit cost, especially when a large model such as GPT-4 is too expensive in production. The second is latency, since smaller models require less computation for each generation. The third is output consistency. A large prompted model may solve a task correctly most of the time while occasionally missing an instruction or formatting requirement. Fine-tuning can make the desired behavior more consistent. Corbitt recommends accepting the early cost of a prompted model while usage is still being tested. After the product is defined and deployment will happen at scale, the savings and quality gains can justify the tuning work.
Starting with a prompted model gives the team a way to learn what the training data should look like before training anything. Corbitt recommends logging each prompt through the application's existing database, observability system, or log provider. The goal is to capture real inputs and the outputs that users receive. If the prompted model already performs well and the goal is mainly to reduce cost or latency, those production examples may be usable directly. If the model makes mistakes, the dataset may need human review or relabeling. A stronger model or a multi-step labeling process can also help create better examples, although it costs more and runs more slowly.
Training infrastructure is no longer the intimidating part
Corbitt breaks the workflow into preparing data, training, evaluating, and deploying. He says the training step often worries people because they expect to manage hyperparameters and infrastructure themselves. In his view, hosted services and open-source libraries have made this part relatively easy. Model architectures and tooling are standardized enough that many teams do not need much custom work. He also says engineers without machine learning or data science experience have successfully produced tuned models in less than an hour of work. The difficult work is more often deciding what the model must do and preparing examples that accurately represent that task.
Evaluation needs a fast loop and a production loop
Corbitt divides evaluation into inner-loop and outer-loop tests. Inner-loop evaluations run quickly and give feedback during model development. At OpenPipe, he describes using an LLM judge with criteria defined by the user to compare tuned outputs against a dataset. Outer-loop evaluation asks whether the system achieves the business result that matters. That measurement depends on the application and may require deploying the model, running an A/B test, or checking production behavior. A model can look better on an example dataset without improving the actual product, so both forms of evaluation are needed before promoting it to production.
Deployment can use hosted or self-hosted inference
Corbitt says deployment has also become less intimidating. A team can self-host a tuned model on its own GPUs with projects that provide an OpenAI-compatible API, or it can use a hosted inference provider. The application can call the model in a familiar way in either setup. His recap is simple: prepare the data, train, evaluate, and deploy. The decision to do this work should come after the team knows the task and has a production reason to change the model's cost, speed, or consistency.
"You have to get the data, you have to actually train the model, you probably should evaluate, and then you have to deploy the model."12:12
Who should watch
You are building an LLM feature and are unsure whether fine-tuning belongs in the first version.
Your prompted model works, but its production cost or latency is becoming difficult to justify.
You need more consistent formatting or instruction-following and want a concrete path from logged production data to a tuned deployment.
