Shipping a new frontier model creates a prompt tax because new capabilities can improve a product while breaking existing prompts and behavior.
2
Orbital migrates faster by making prompts leaner for system 2 models, using feature flags, having models help rewrite prompts, and acting on user feedback within minutes or hours.
3
Andrew Thompson argues that teams should ship new models progressively and accept uncertainty rather than waiting for a complete eval system.
Summary
Andrew Thompson describes the prompt tax that comes with adopting rapidly improving AI models. At Orbital, an agentic product for real-estate due diligence grew from GPT-3.5 to newer system 1 and system 2 models while its domain-specific prompt library grew to more than 1,000 prompts. Each migration can unlock new product behavior, but it can also regress existing features and create anxiety among users and domain experts. Orbital has dealt with this by favoring prompting over fine-tuning, relying heavily on real-estate lawyers, using human testing instead of a comprehensive eval system, and rolling out model changes progressively. Thompson recommends removing detailed instructions and repeated constraints when moving to reasoning-oriented models. He also describes using the new model to help migrate old prompts and routing feedback directly to domain experts and AI engineers. His conclusion is to buy into new model capabilities now, then decide how much prompt tax to pay as problems appear.
Rapid model progress creates a recurring prompt tax
Andrew Thompson compares frontier model releases to having a birthday almost every month. New functionality can be added to an application, but the probabilistic behavior of the models can also produce regressions and unintended consequences. He describes the prompt tax as the work required to understand what a new model improves, what it breaks, and how much existing prompting must change. This differs from ordinary technical debt because teams often want to upgrade prompts immediately to use newly available capabilities. The resulting tension is between shipping those capabilities and protecting behavior that users already rely on.
Orbital applies agentic systems to real-estate due diligence
Orbital automates parts of the legal due-diligence process for property transactions. Its software reads large collections of documents, finds items such as lease dates and annual rent, and compiles the results into a report that a licensed real-estate lawyer can review. In the demonstration, a deed and lease totaling about 100 pages are OCRed, divided into subtasks, and processed by multiple agentic systems with multiple LLM calls. The final report includes clickable citations that let the lawyer return to the source documents. Thompson says the company has grown from less than a billion tokens per month to almost 20 billion tokens per month over 18 months.
Orbital chose fast prompt iteration over fine-tuning
Orbital optimized for prompting because prompts could be changed quickly after user feedback and pulled into the agentic system in real time. This helped the team move quickly while it was finding product-market fit and continues to support rapid feature work. Real-estate lawyers at Orbital write many of the domain-specific prompts, turning decades of legal experience into instructions for the system. AI engineers own the system prompts that guide tool selection and use. Thompson also says the company reached its current scale mostly through human judgment, which he calls using vibes over evals, with domain experts testing releases, recording some results in spreadsheets, and looking for regressions.
Moving to system 2 models means removing instructions
Thompson says system 1 models such as GPT-4o often required detailed instructions about how to complete a task, along with repeated reminders of those instructions. When Orbital moved toward system 2 models such as o1 preview, the team removed much of that detail and focused on stating the objective. These models work better when they have fewer constraints and time to reason through the context and form a plan. System 1 models still have uses because they are often cheaper and faster. Their thought tokens can be shown to users who want an explanation, or inspected by engineers when debugging a failure.
Progressive rollout reduces the risk of model upgrades
Orbital applies feature-flag thinking to model migrations. It can test an upgrade internally, release it to a limited group of users, fix problems from their feedback, and expand the rollout. Thompson warns that users often feel more anxious about the new system simply because it is new. Existing problems are familiar, while the new model may expose unknown problems. Announcing a migration can therefore make people look harder for failures. Progressive delivery gives the team a way to distinguish real regressions from heightened attention while still moving toward a wider release.
Orbital's team mantra is to bet on the model. Thompson says teams should avoid building software around features that will exist only in the present and should instead consider where models may be in three, six, or twelve months. If models become smarter, cheaper, faster, and more capable, a product feature designed with that direction in mind can improve over time. The mantra gives the team a short way to challenge present-focused product decisions. It asks whether a feature would make more sense if the team planned around future model capabilities rather than today's limitations.
Fast feedback can pay back prompt tax in minutes or hours
Orbital sends in-product feedback directly to a domain expert. That person evaluates the issue, identifies the prompt that needs to change, and ships the update to production. The fix can help the user who reported the problem and other users on the same system. Thompson says this sometimes takes minutes or hours instead of the days or weeks associated with more traditional bug cycles. He sees this feedback loop as a practical way to keep shipping while the team lacks a comprehensive evaluation system. The approach depends on getting user feedback quickly into the hands of people who can change the prompts.
A full eval system may become too expensive to maintain
Thompson is unsure whether evals will scale as Orbital's product surface grows. A real-estate legal answer must satisfy several dimensions, including factual correctness, style, conciseness, and citation correctness. The system also has to cover the many edge cases and use cases that lawyers provide. Evaluating all of those cases could become prohibitively expensive and slow, and the number of evals might grow faster than the product team can maintain them. His current alternative is progressive delivery, with the rollout expanding according to the amount of feedback the team can handle.
"Ultimately, shipping product at the frontier has this constant tension between the opportunities that new AI models can give versus the risks that you could introduce either regressions or unintended consequences to your products."01:19
Who should watch
You are building an agentic product whose prompts and model behavior change whenever a frontier model is released.
Your product depends on domain experts who can review model output and turn user feedback into prompt changes.
You are deciding whether to wait for a broad eval suite or ship a new model through a controlled rollout.
