CoreWeave is building one inference platform for serverless, provisioned throughput, and dedicated deployments across different model sizes and workload types.
2
KV-cache-aware routing reduces repeated prefill work, especially for agentic requests where 80 to 90 percent of the input can repeat across turns.
3
The platform combines scheduling, cache offload, quantization, and customer-specific speculative decoding to improve price performance.
Summary
Sitanshu Gupta describes the inference platform he is helping build at CoreWeave after four months at the company. The platform supports serverless usage, provisioned throughput for customers with known traffic patterns, and dedicated deployments billed by GPU hour. Gupta organizes the system around four workload shapes: agentic and chat requests, real-time voice and video, and flexible batch work. Their different latency and throughput needs can be fitted across the same infrastructure over time. The router prioritizes KV-cache locality before falling back to the least-loaded deployment, including across heterogeneous GPUs and zones. Chat-session caches can be moved to high-bandwidth storage instead of being discarded. Gupta also discusses four-bit quantization and asynchronous training of speculative decoders on customer data. The talk is a practical account of how consumption models shape the platform and how repeated prefill work affects both architecture and pricing.
The platform starts with three ways customers consume inference
Gupta describes serverless inference as an API and UI where customers manage neither hardware nor orchestration and pay per token. Dedicated inference gives customers more control over the hardware, model deployment, engine, and performance settings while CoreWeave provides the service and orchestration layers. Serverless also includes provisioned throughput. A customer with a known traffic profile can have capacity carved out for them without choosing the underlying hardware, reducing the risk of noisy neighbors while keeping token-based billing.
Agentic requests have the same shape as chat, but a tighter latency requirement
Agentic and chat workloads usually have long input sequences and short outputs. In chat, a person reads the response before sending another turn, so the gap between turns gives the system more time. Agentic turns arrive with little or no human pause, which makes them much more latency-sensitive. Voice and video are also real-time workloads with strict latency needs. Batch work is different: some customers can submit 10 to 12 hours of work and accept processing over a much longer window.
The scheduler fits different workload shapes onto the same capacity
Gupta compares capacity planning to playing Tetris across time. A platform has to place real-time and flexible workloads so the underlying infrastructure is used well without violating service levels. A dedicated deployment can serve a customer's real-time traffic during the US daytime, then run batch work from evening through the night. The API lets the customer specify when capacity should scale up or down, allowing the platform to open that capacity for scheduled batch processing.
The request path separates control-plane work from model execution
Requests can come from applications, notebooks, or agents and first reach a gateway. The control plane handles authorization, rate limits, usage tracking, billing support, and observability. From there, requests enter serverless or dedicated deployments and are sent through inference engines and different generations of NVIDIA GPUs. Serverless routing differs for pay-as-you-go, provisioned throughput, and multi-tenant deployments. Dedicated customers use private gateways and pay by GPU hour rather than by token.
KV-cache locality comes before least-loaded routing
Gupta says 80 to 90 percent of an agentic request's input can repeat across requests, depending on the customer. Recomputing that repeated prefill wastes work because prefill is compute-bound and expensive. The router therefore tries to send a request to the deployment that already has the relevant KV cache. When cache locality cannot decide the placement, it falls back to the least-loaded option. The same priority is used across heterogeneous capacity in different zones and regions.
Dedicated customers can choose how the model is split and served
A dedicated customer receives an isolated gateway and can choose the inference engine, including VLM, SGLang, or TensorLM, according to Gupta. The customer can decide whether to separate prefill and decode, although that split is not useful for every workload. They can also use one deployment that scales across the reserved cluster or run several models and deployments. The platform exposes these choices while keeping the routing logic for cache-heavy requests.
Long pauses between chat turns create a different cache problem. If the system evicts a conversation's KV cache, the next request must repeat the prefill and takes longer. Gupta says the platform can offload KV cache to high-bandwidth storage instead. When the next request for that conversation arrives, the cached prefill can be loaded back into GPU memory. He mentions LMCache and Mooncake as external examples, while saying CoreWeave uses its own approach.
Quantization and customer-trained speculators are current performance levers
Gupta names quantization to NVFP4 and speculative decoding as two major areas of work. If a customer supplies a dataset, CoreWeave can train speculative models for that customer's data to improve acceptance lengths and raise output throughput. The data collection, training, and deployment happen asynchronously. He also mentions prefill and decode disaggregation and the choice of parallelization degrees and strategies as other performance decisions.
"If we completely evict whatever we had in our particular conversation then the next time we ask a question in that same chat it's going to take a little bit longer."11:44
Who should watch
You are designing an inference service that must support both hands-off serverless users and customers who control their own deployments.
Your workload includes long repeated prompts, agentic turns, or chat sessions where prefill cost and cache retention affect latency.
You have dedicated GPU capacity that sits idle at certain times and want to schedule batch work without disrupting real-time traffic.
