Most teams should begin with a prompted model because it gives them fast iteration and reveals what users actually need.
2
Fine tuning becomes useful when cost, latency, or inconsistent quality makes the prompted approach hard to operate.
3
A practical fine-tuning workflow is to capture production data, improve the dataset, train, evaluate in inner and outer loops, and deploy.
Summary
Kyle Corbitt argues that fine tuning can move the quality-cost frontier outward. A fine-tuned model can produce better results at the same inference budget than a larger prompted model, although it takes preparation and reduces flexibility outside the task it was trained for. Most teams should first use a prompted model such as GPT-4 while they discover the product shape and collect real user inputs. The triggers for moving to fine tuning are high unit cost, excessive latency, or quality that is usually correct but inconsistent. Corbitt then explains a four-step workflow: capture prompts and outputs, improve the training data, train the model, evaluate it, and deploy it. He separates fast inner-loop evaluations from business-specific outer-loop tests in production. His advice is practical and honest about the trade-off: fine tuning is easier than many engineers expect, but it is only useful once the task and data distribution are understood.
Corbitt begins with a graph that compares output quality with tokens per dollar. Larger, more expensive models usually produce better zero-shot results, so teams choose a model based on either the quality they need or the spending limit they have. Prompting techniques that use more inference-time compute can push quality higher, but they also add cost and latency. Fine tuning creates another option. In his internal evaluations across four customer datasets, fine-tuned versions of Llama 3 70B and Llama 3 8B performed better than their prompted versions and better than the base models on the graph. This changes the available trade-off for a given inference budget.
Fine tuning has an upfront cost because a team must collect data and verify that the model works. It also makes the model less flexible. The model becomes very good at a specific task, which means inputs outside that task may not receive the same performance improvement. Corbitt says teams need to understand their task before training and should avoid sending unrelated, out-of-domain requests to the resulting model. This is why fine tuning is often the wrong first step. A prompted model lets a team change behavior quickly while the product and its real inputs are still changing.
Cost, latency, and consistency are the signals to move up
Corbitt gives three practical reasons to move from prompting to fine tuning. The first is unit cost, especially when a large model such as GPT-4 is too expensive in production. The second is latency, since smaller models require less computation per generation. The third is consistency. A prompted model may solve a task correctly most of the time while occasionally missing an instruction or returning the wrong format. Fine tuning is well suited to making the desired output more consistent. Corbitt recommends accepting the early cost of a prompted model while the product is being tested, then training once the product shape and production behavior are clear.
Production prompting reveals the training distribution
Starting with a prompted model provides information that is hard to know in advance: what users actually send and what the system needs to handle. Teams can log prompts and outputs in the database, observability system, or logging provider they already use. This gives them a picture of the input distribution before they train. If the prompted model already produces good outputs, that production data may be enough to create a useful training set. If it makes mistakes, the team may need human review, relabeling, or a stronger model to produce better labels.
A stronger labeling loop can improve training data
When the base prompted model makes errors, Corbitt describes using multiple model passes to generate or improve labels. His mixture-of-agents example uses several iterations and remains automated, which can address some reliability problems in the training data. The trade-off is that labeling becomes slower and more expensive. Since the process is used to create a dataset rather than to answer every production request, Corbitt says the additional work can still be manageable. The important decision is whether the outputs used for training are good enough to teach the specialized model the behavior the application needs.
Training infrastructure is now mostly straightforward
Corbitt says many engineers expect hyperparameters and infrastructure to be the hardest part of fine tuning. In his view, that part is mostly solved. Hosted services can run training with a simple workflow, and open-source libraries are available for teams that want to manage more of the process themselves. Standardized model architectures and better tooling reduce the amount of custom work required. The difficult part is usually understanding the task and preparing reliable data, rather than operating the training machinery.
Corbitt divides evaluation into an inner loop and an outer loop. Inner-loop tests run quickly and provide feedback during development. OpenPipe uses language-model judges with criteria defined by the user to compare fine-tuned outputs against a dataset. Outer-loop evaluation measures what matters for the business and the specific use case. It may require deploying the model or running an A/B test in production to determine whether the model changes the desired outcome. A model that looks better on a dataset is not automatically better for the application.
Deployment can use hosted or self-managed inference
The final step is deployment, which Corbitt says has become less intimidating. Teams can self-host a model on their own GPUs using projects that provide an OpenAI-compatible API, or they can use hosted inference providers. The workflow he presents is therefore four steps: prepare data, train, evaluate, and deploy. Fine tuning is not an early requirement for every application. It becomes a practical choice after the team understands its inputs and has a specific cost, latency, or consistency problem to solve.
