Inference is a tensor-to-tensor workload whose main costs come from moving model weights through memory and performing the required math.
2
Proprietary models currently offer the highest capabilities and affordable hosted access, while open models provide more control and room for parallel development.
3
Running inference on user devices can remove serving costs and network delay, but it introduces tight limits around memory, power, heat, compatibility, and telemetry.
Summary
Charles Frye explains what happens after an application calls a foundation model API. Generative inference converts human-readable inputs into tensors, runs them through a neural network, and converts the result back into text, images, or sound. The neural network is the expensive part, so teams must decide whether to use a proprietary service, run an open model, or serve inference themselves. Proprietary models currently lead on capability and are often inexpensive to use, while open models offer control and support parallel experimentation. Frye expects both categories to coexist, with open models gaining ground when capability requirements stop increasing quickly. He then compares execution on phones, desktops, browsers, public clouds, serverless GPUs, and owned hardware. The engineering limits are concrete: memory bandwidth, model size, latency, power, heat, batching, cold starts, and GPU utilization. Self-hosting can reduce long-term costs, but it requires specialized infrastructure and careful profiling.
Inference turns human data into tensors, then turns model output back into human data
Frye describes generative inference as a data-to-data function whose inputs and outputs are understandable to people, such as text, images, and sounds. Internally, tokenizers convert text into arrays of numbers, neural networks map tensors to tensors, and another step converts tensors back into text or other outputs. The tensor-to-tensor operation is the bottleneck. It consumes most of the compute and memory, and it receives most of the engineering attention. Frye uses a robot given a restaurant image and asked what steps would make it useful as an example of the full path from an input image to generated instructions.
Proprietary models currently trade control for higher capability and simple access
Frye frames model selection as a build-versus-buy decision. He says proprietary models are currently the most capable, so teams often start with them to prove that an application works, then consider a cheaper model later. Hosted pricing can also be affordable compared with running the same workload independently. The tradeoff is control. Providers can remove access to log probabilities because detailed outputs may help users reverse-engineer the model. Enterprise offerings add governance, security, compliance, service-level agreements, and higher rate limits. Fine-tuning may help with style, but Frye says it is not a good way to inject new information and usually costs more at inference time.
Open models gain from parallel experimentation, but apparent quality can be misleading
Open models are less capable in Frye's comparison, but they are catching up and are easier to modify. He connects their progress to the open-source pattern of many people making small improvements that others can reuse. He also warns that model licenses such as Llama 2's do not meet the usual definition of open source because they restrict use and distribution. Frye demonstrates why leaderboard judgments can fail: a fine-tuned model's answer may look polished and authoritative while being wrong. In his example, a GPT-4 answer correctly explains actor-critic methods, while a more stylistically appealing answer from an open fine-tuned model contains false claims.
Open and proprietary models are likely to coexist as different software-stack components
Frye compares foundation models with operating systems and databases. Closed and open systems have coexisted in both areas, serving organizations with different needs. He expects a similar split for models. Open models may dominate if users' capability requirements reach a point where a model is simply good enough, because open projects can then catch up without paying the cost of continually building larger teams and data centers. Proprietary models can retain an advantage if users continue demanding the next level of capability. The answer depends on whether requirements saturate. Frye does not present a single winner.
End-user inference removes network cost and can meet latency needs that a remote API cannot
Running inference on a phone, robot, wearable, desktop, or browser can reduce serving costs to zero and avoid repeated network delays. Frye cites the roughly 150-millisecond round trip between California and Europe as an example of the delay that can consume an interactive system's budget. Network execution is especially unsuitable for a moving system that needs responses in tens of milliseconds. Local execution gives up control, telemetry, hardware consistency, and the ability to support only one controlled environment. Phones and small computers impose severe limits on RAM, VRAM, power, and heat. Frye gives an A100's 300 watts versus a Jetson Nano's 10 watts as an example of the power gap.
Quantization and sparsity reduce model size, but the useful gains often require low-level engineering
Frye explains that model parameters commonly start at two bytes each, so a seven-billion-parameter model occupies about 14 GB before other requirements are counted. Quantization aims to reduce parameters to one byte, half a byte, or even a single bit. He says progress often stalls around half a byte, and making the hardware use those savings may require hand-written assembly. Sparsity can also remove weights near zero, but neural networks need unstructured sparsity, while widely available tools tend to support structured sparsity. Taking advantage of unstructured sparsity may require hand-tuned CUDA kernels. Distillation and quantization have already helped fit language-model capabilities onto mobile hardware.
Cloud and serverless inference exchange infrastructure control for usage-based cost
Inference-as-a-service providers make it easy to run proprietary models and open models such as those available through Replicate. They can lower costs by sharing infrastructure across users, but that requires similar usage patterns and limits customization. Public clouds offer more control and can be cheaper than specialist services, although GPU support varies and long-term use can become expensive. Serverless systems add scale-to-zero pricing, so a low-traffic demo uses no resources when nobody calls it. Their weakness is control over scaling and cold-start latency. Loading weights into accelerator memory can take 30 seconds, one minute, or three minutes when the model is very large.
GPU inference is often limited by memory movement, so batching and profiling determine performance
Frye reduces the GPU workload to two operations: loading numbers from memory and doing math on them. Memory is slow while arithmetic is fast, so inference needs to reuse each loaded weight across many computations. GPUs fit this pattern because they provide parallel linear algebra and high memory bandwidth. Frye says an A100 can perform 312 teraflops for two-byte operations while providing 1.5 terabytes per second of memory bandwidth, which creates a strong incentive to batch requests. At small batch sizes, memory dominates, and at larger batch sizes, computation becomes more relevant. Profilers and traces can expose wasted CUDA launches, idle GPU time, and memory problems that aggregate statistics miss.
Self-hosted inference makes GPU, container, and orchestration choices depend on one another
Containerizing inference is easier than it used to be, but model weights can make images extremely large. The CUDA driver and GPU type also affect application decisions such as batching and the point where a workload changes from memory-bound to compute-bound. Containers virtualize only part of the environment, so the host driver can affect which CUDA features and kernels are available inside the container. Kubernetes adds another layer to the problem, especially for heterogeneous clusters with different GPU, CPU, and processor architectures. Teams can build these systems themselves with tools such as Ray Serve or Seldon Core, or choose managed offerings such as Vertex AI, Bento Cloud, Seldon, or Anyscale.
"The closest thing to an answer that I have is that if capabilities requirements saturate, if people no longer want the absolute smartest model out there, they just want a model smart enough for XYZ, then open models will probably catch up and then start to dominate."34:21
Who should watch
You are deciding whether to call a proprietary model API, deploy an open model, or build an inference platform.
Your application needs low latency on a phone, robot, browser, or other device with limited memory and power.
You are sizing GPUs or debugging self-hosted inference and need to understand batching, memory bandwidth, CUDA, containers, and cold starts.
