# Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads

Sitanshu Gupta, CoreWeave | AI Engineer World's Fair 2026 | 15:22

Source: https://www.youtube.com/watch?v=cQQbJqvZkpo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/vertical-mobility-inference-from-mvp-to-trillion-parameter-workloads
Published: 2026-09-19
Tags: caching, deployment, inference, quantization

## TL;DR
- CoreWeave is building one inference platform for serverless, provisioned throughput, and dedicated deployments across different model sizes and workload types.
- KV-cache-aware routing reduces repeated prefill work, especially for agentic requests where 80 to 90 percent of the input can repeat across turns.
- The platform combines scheduling, cache offload, quantization, and customer-specific speculative decoding to improve price performance.

## Summary
Sitanshu Gupta describes the inference platform he is helping build at CoreWeave after four months at the company. The platform supports serverless usage, provisioned throughput for customers with known traffic patterns, and dedicated deployments billed by GPU hour. Gupta organizes the system around four workload shapes: agentic and chat requests, real-time voice and video, and flexible batch work. Their different latency and throughput needs can be fitted across the same infrastructure over time. The router prioritizes KV-cache locality before falling back to the least-loaded deployment, including across heterogeneous GPUs and zones. Chat-session caches can be moved to high-bandwidth storage instead of being discarded. Gupta also discusses four-bit quantization and asynchronous training of speculative decoders on customer data. The talk is a practical account of how consumption models shape the platform and how repeated prefill work affects both architecture and pricing.

## Key ideas
### The platform starts with three ways customers consume inference
[01:13](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=73s)
Gupta describes serverless inference as an API and UI where customers manage neither hardware nor orchestration and pay per token. Dedicated inference gives customers more control over the hardware, model deployment, engine, and performance settings while CoreWeave provides the service and orchestration layers. Serverless also includes provisioned throughput. A customer with a known traffic profile can have capacity carved out for them without choosing the underlying hardware, reducing the risk of noisy neighbors while keeping token-based billing.

### Agentic requests have the same shape as chat, but a tighter latency requirement
[03:10](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=190s)
Agentic and chat workloads usually have long input sequences and short outputs. In chat, a person reads the response before sending another turn, so the gap between turns gives the system more time. Agentic turns arrive with little or no human pause, which makes them much more latency-sensitive. Voice and video are also real-time workloads with strict latency needs. Batch work is different: some customers can submit 10 to 12 hours of work and accept processing over a much longer window.

### The scheduler fits different workload shapes onto the same capacity
[04:53](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=293s)
Gupta compares capacity planning to playing Tetris across time. A platform has to place real-time and flexible workloads so the underlying infrastructure is used well without violating service levels. A dedicated deployment can serve a customer's real-time traffic during the US daytime, then run batch work from evening through the night. The API lets the customer specify when capacity should scale up or down, allowing the platform to open that capacity for scheduled batch processing.

### The request path separates control-plane work from model execution
[05:15](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=315s)
Requests can come from applications, notebooks, or agents and first reach a gateway. The control plane handles authorization, rate limits, usage tracking, billing support, and observability. From there, requests enter serverless or dedicated deployments and are sent through inference engines and different generations of NVIDIA GPUs. Serverless routing differs for pay-as-you-go, provisioned throughput, and multi-tenant deployments. Dedicated customers use private gateways and pay by GPU hour rather than by token.

### KV-cache locality comes before least-loaded routing
[07:17](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=437s)
Gupta says 80 to 90 percent of an agentic request's input can repeat across requests, depending on the customer. Recomputing that repeated prefill wastes work because prefill is compute-bound and expensive. The router therefore tries to send a request to the deployment that already has the relevant KV cache. When cache locality cannot decide the placement, it falls back to the least-loaded option. The same priority is used across heterogeneous capacity in different zones and regions.

### Dedicated customers can choose how the model is split and served
[08:48](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=528s)
A dedicated customer receives an isolated gateway and can choose the inference engine, including VLM, SGLang, or TensorLM, according to Gupta. The customer can decide whether to separate prefill and decode, although that split is not useful for every workload. They can also use one deployment that scales across the reserved cluster or run several models and deployments. The platform exposes these choices while keeping the routing logic for cache-heavy requests.

### KV cache can move to storage between chat turns
[11:05](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=665s)
Long pauses between chat turns create a different cache problem. If the system evicts a conversation's KV cache, the next request must repeat the prefill and takes longer. Gupta says the platform can offload KV cache to high-bandwidth storage instead. When the next request for that conversation arrives, the cached prefill can be loaded back into GPU memory. He mentions LMCache and Mooncake as external examples, while saying CoreWeave uses its own approach.

### Quantization and customer-trained speculators are current performance levers
[12:52](https://www.youtube.com/watch?v=cQQbJqvZkpo&t=772s)
Gupta names quantization to NVFP4 and speculative decoding as two major areas of work. If a customer supplies a dataset, CoreWeave can train speculative models for that customer's data to improve acceptance lengths and raise output throughput. The data collection, training, and deployment happen asynchronously. He also mentions prefill and decode disaggregation and the choice of parallelization degrees and strategies as other performance decisions.

## Notable quotes
- "The platform needs to be capable enough of having the workload getting distributed across various different generations of these GPUs." (05:58)
- "The priority order that we typically take is first KV cache locality and then the least loaded fallback." (09:53)
- "If we completely evict whatever we had in our particular conversation then the next time we ask a question in that same chat it's going to take a little bit longer." (11:44)
- "Two of the biggest levers that we've been working with are quantization to NVFP4 and speculators." (12:52)

## Tools & references mentioned
- CoreWeave
- AWS Annapurna Labs
- SambaNova
- NVIDIA GPUs
- VLM
- SGLang
- TensorLM
- LMCache
- Mooncake
- NVFP4
- Artificial Analysis
- OpenRouter
- Weights & Biases
- Fireworks

## Who should watch
- You are designing an inference service that must support both hands-off serverless users and customers who control their own deployments.
- Your workload includes long repeated prompts, agentic turns, or chat sessions where prefill cost and cache retention affect latency.
- You have dedicated GPU capacity that sits idle at certain times and want to schedule batch work without disrupting real-time traffic.

## Related talks

- [Hacking the Inference Pareto Frontier](https://aietalks.com/talks/hacking-the-inference-pareto-frontier) (Kyle Kranen, NVIDIA, 20:25)
- [Deep dive on LLM Inference at Scale](https://aietalks.com/talks/deep-dive-on-llm-inference-at-scale) (Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher, 1:28:12)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)
- [AI Engineering 201: Inference](https://aietalks.com/talks/ai-engineering-201-inference) (Charles Frye, Full Stack Deep Learning, 1:43:16)
- [System Design for Next-Gen Frontier Models](https://aietalks.com/talks/system-design-for-next-gen-frontier-models) (Dylan Patel, SemiAnalysis, 18:29)