Maturing AI products often move toward fine-tuning because their business logic, cost targets, and latency needs become more specific than a frontier API can support.
2
Three signs that fine-tuning may be worthwhile are paying more for the API than customers pay you, plateauing evals, and latency or throughput requirements that shared endpoints cannot meet.
3
An existing agent harness, evaluation process, and collected usage data can provide much of what a team needs to begin supervised fine-tuning or reinforcement learning.
Summary
Benjamin Cowen argues that frontier APIs are a starting point for AI products, but specialization eventually creates reasons to control the model more directly. Frontier labs optimize for broad performance across many tasks. An application team may instead need a model that follows its own business logic, meets a particular latency target, or costs less to operate. Cowen describes fine-tuning as a middle ground between prompt-only APIs and owning the entire training and serving stack. Open-source libraries and serverless compute can provide algorithm control without requiring teams to build a large cluster. He gives practical signals for starting: API costs that exceed customer revenue, evals that have stopped improving, or requirements shared endpoints cannot meet. Teams with an agent harness and mature evaluation data may already have the foundation for reinforcement learning. Cowen also covers short Python training programs, on-demand hyperparameter searches, large-scale RL rollouts, and serving models with tools such as vLLM and Triton.
Frontier APIs are fast to start but cannot encode every application's business logic
Cowen places frontier APIs at one end of a model spectrum. They let teams build quickly and perform well across many tasks, but customization largely stops at prompt engineering. Speaking like a caveman may reduce token use, but Cowen says that approach will not scale if a startup grows by 100 or 1,000 times. Enterprise customers can also require specific latency or throughput, while an application may have a custom metric tied to its business logic. A shared endpoint offers little control over those constraints.
Fine-tuning used to require a large jump into infrastructure ownership
Traditionally, fine-tuning meant moving from an API to a much more demanding training setup. Training and production workloads have different scaling and compute needs, so teams needed a large cluster and had to isolate training resources from production resources. Infrastructure engineers, AI engineers, or scientists then spent time managing that stack. Cowen describes this as a powerful option that also brings responsibility for the whole system. He presents serverless compute and newer cloud providers as a middle ground.
A specialized model can optimize for the application's business logic
Cowen points to companies reporting better results after fine-tuning. He says Intercom is beating its frontier API at one fifth the cost, while Pinterest has reported orders-of-magnitude gains. Decagon summarizes the difference in goals: frontier labs want models that win across everything possible, while an application team wants its model to win at the service it provides customers. Fine-tuning gives the team control over the training algorithm while preserving faster iteration than building the entire infrastructure stack itself.
Product maturity creates a practical decision point for custom models
Cowen's view is that a differentiated product is already becoming domain-specific, even if the team has not decided to train a model. He names three signals that the decision may be near. A company may still be paying more for its API than customers pay it after trying prompt techniques such as caveman mode. Its evals may have plateaued. Or its latency and throughput requirements may exceed what a shared endpoint can provide. These conditions suggest that customized inference or fine-tuning could improve the product's economics or performance.
Training requires useful data and mature evaluations
Cowen applies the familiar garbage-in, garbage-out rule to model training. Teams that have not collected data or developed mature evals are probably not ready to train, because they lack the material needed to judge and improve the model. He also says that teams which have built a product have probably already touched much of the required foundation. Evaluating what works, recording what fails, and collecting those examples can turn normal product development into training data.
An agent harness can become the foundation for reinforcement learning
If a team has built an agent harness, Cowen says it has what it needs for a new model to learn through reinforcement learning how to provide the service. The harness produces interactions that can be evaluated, while the product process supplies information about what works and what does not. Cowen contrasts current tooling with earlier practice, when machine learning teams might tape gradients by hand or implement linear algebra themselves. Open-source libraries now make many training workflows accessible without infrastructure specialists.
Serverless compute can fan out training experiments on demand
Cowen says supervised fine-tuning can fit in 300 lines of Python once the data is curated and a team has access to a serverless platform. Serverless resources are also useful for hyperparameter tuning. A team can launch many containers when needed, stop runs that look unpromising, and avoid treating every minute on a fixed cluster as precious. The same model applies to reinforcement learning, where training depends on many parallel evaluations called rollouts.
RL rollouts and model serving can scale separately from the training code
Cowen describes customers scaling to 50,000 or 100,000 sandboxes for reinforcement learning rollouts. Modal provides unified APIs for sandboxes and GPU containers, which lets the training model practice through many parallel environments. After training, the model still needs to be served. Cowen names vLLM, SGLang, Triton Inference Server, and custom Python workflows as possible serving approaches. Serverless platforms can autoscale the serving layer as traffic changes.
Teams should prepare for training before they need it
Cowen does not tell every team to train a model immediately. He says training may become appropriate in six months or a year, so teams should decide what evidence will tell them the time has arrived. Preparing means collecting useful product data and developing evals before the decision is urgent. This makes training a planned option rather than a distant project that requires starting from scratch.
"If you have built an agent harness, then you have what you need to have a new model learn through reinforcement learning how to provide your service."07:06
Who should watch
Your team is relying on a frontier API but API spend, latency, or throughput is starting to constrain the product.
You have an agent harness, evaluation process, and growing interaction data, and want to know whether they are enough to begin training.
You are weighing a custom model against owning a dedicated training and inference cluster.
