Enterprise AI adoption has moved from experimentation to production, but closed models are now running into limits around task quality, latency, cost, and differentiation.
2
Open models become useful when companies need to do better on a specific task, control inference economics, or run latency-sensitive applications such as voice systems.
3
Running an open model in production requires much more than choosing a model, connecting it to GPUs, and deploying an inference server.
Summary
Amir Haghighat describes how enterprise AI adoption has changed through conversations with more than 100 companies. In 2023, many enterprises used dedicated OpenAI deployments for experimentation. In 2024, production use cases appeared. By 2025, Haghighat was seeing companies question whether closed frontier models would remain the right foundation for every workload. The reasons were specific: custom models could outperform generic APIs on tasks such as medical document extraction, latency mattered for voice applications, agentic systems could generate many inference calls per user action, and companies wanted more control over their economics and differentiation. Moving to open models creates a harder engineering problem. Teams must optimize both models and infrastructure, handle failures and traffic spikes, manage observability and audits, and meet reliability targets without excessive overprovisioning. Haghighat presents this as a build-or-buy decision for an inference platform layer.
Enterprise adoption matters because large companies determine how widely AI changes work
Haghighat argues that enterprise adoption is a useful test for whether AI's impact will become large. Enterprises have extensive reach and resources, so slow adoption would delay the broader shift people expect from AI. He bases his view on more than 100 conversations over the previous two years, ranging from public software companies to Fortune 50 beverage companies. He also distinguishes between buying vertical products and building with AI. Sales, marketing, and customer-service tools are already being adopted, but he expects more value once enterprises become comfortable building their own systems, as they did with software infrastructure companies such as Snowflake, Databricks, and Datadog.
Most enterprises begin with dedicated deployments of OpenAI or Anthropic models
The common starting point is a closed model from OpenAI or Anthropic, deployed through Azure or AWS in an environment dedicated to the enterprise. Security and privacy requirements make this arrangement easier to approve. Existing predictive machine-learning teams then become AI teams and build applications on top of an API. Haghighat says this approach is attractive because it is easy to use, and companies will keep using it when it works. He describes a progression from experimentation in 2023, to production use cases in 2024, to visible doubts about relying on closed frontier models indefinitely in 2025.
The shift toward open models is driven by workload fit rather than fear of vendor lock-in
Haghighat says vendor lock-in is not what enterprise buyers usually mention. OpenAI, Anthropic, and Google are somewhat interoperable, and moving between them may require prompt tuning rather than a complete rebuild. Compliance, privacy, and security are also being handled through model providers and cloud providers inside dedicated virtual private clouds. The pressure comes from the work itself. Health plans have used labelled data to build better systems for extracting procedure codes, diagnosis codes, and prescriptions from medical documents. Healthcare transcription systems can also benefit from models trained to understand medical language instead of using a generic API.
Latency and agentic workloads expose limits in the standard API model
Haghighat says shared model services are generally optimized for high throughput and high query volume, which can come at the expense of latency. This matters in AI voice and phone applications, where time to first audio and time to the first sentence affect the experience. Agentic applications create a separate cost problem. A single user action can trigger 50 inference calls, so falling token prices do not automatically make the total system economical. Some enterprises are deciding that paying for compute directly and running models themselves could cost less than paying per token while also giving the provider a margin.
Some CIOs want control of the model because generic models do not differentiate their companies
Haghighat describes this motivation as less concrete than quality, latency, or cost, but says he hears it more often. If an enterprise and its competitors use the same frontier models, executives ask where their advantage comes from. Bringing models in-house can create differentiation at the AI layer, in addition to differentiation in the workflow or application. The goal is not necessarily to train a general model that beats OpenAI. It can mean adapting, fine-tuning, or distilling an open model for a company's own tasks and data.
Production inference requires a platform, not just a model and a GPU
Once a company chooses an open model, it moves from calling a simple API to building inference infrastructure. Haghighat rejects the idea that selecting a model, adding a framework such as SGLang or TensorRT-LLM, and attaching GPUs automatically produces production inference. Teams need to make the system scale, keep latency under control, and let engineers ship without waiting to hire a separate group to build everything. The engineering work spans model optimization, infrastructure, reliability, lifecycle management, observability, controls, and audits.
Meeting latency targets requires decisions at both the model and infrastructure layers
For latency-sensitive systems, Haghighat says teams must optimize the model and the infrastructure together. Model-level choices can include speculative decoding, Medusa heads, Eagle 3, and multi-token prediction. These techniques change quickly, so a team needs people who can follow current research and apply it in production. Infrastructure choices matter as well. Prefix caching and disaggregated serving can help with large, similar prompts, especially in agentic workloads. They also affect time to first token and the reliability of tail-latency targets such as P99.
Reliability and burst handling can determine whether self-hosting is economically practical
Mission-critical inference needs more than a nominal deployment. Haghighat asks how teams will reach four nines of availability when a GPU fails, an inference server crashes, or tail latency rises while services restart. They also need to handle sudden traffic increases without waiting too long for replicas. He cites a large enterprise that needed eight minutes to bring up a new replica of a model, which could make tail latency unacceptable during a spike. Overprovisioning may reduce those risks, but it can damage the unit economics that motivated self-hosting. That leaves enterprises with a build-or-buy decision for the inference platform layer.
"I know for a fact that this is not true. I wish it was true, but I know for a fact that there's a lot more that goes into making inference, especially mission critical inference, work well inside of your company."11:31
Who should watch
You are deciding whether a closed model API is still suitable for a production enterprise workload.
Your team is evaluating open models for medical extraction, transcription, voice, or agentic applications where quality, latency, or inference cost matters.
You are building or buying an inference platform and need a candid list of the operational problems that appear after the first deployment.
