Local inference focuses on fitting a model onto available hardware, while data center inference focuses on reducing latency after the model works.
2
Training dedicated components for inference is blurring the boundary between model training and serving optimization.
3
Recent progress in speculation has moved from small draft models to trained and diffusion-based drafters, with continuous retraining offering further gains when its operational costs are acceptable.
Summary
Philip Kiely reviews developments in inference engineering since the publication of his book in February 2026. He separates local inference, where engineers compress models to fit available hardware, from data center inference, where the goal is to reduce serving latency and coordinate system resources. The talk focuses on quantization, KV-cache management, and speculative decoding. TurboQuant can halve KV-cache storage and increase effective bandwidth, but its extra decode computation makes it a poor fit for many production data center workloads. Kiely describes KV compaction through a learned bottleneck that turns a full cache into a shorter representation. He then follows speculation from small draft models to Eagle 3, DFlash, and the newer DSpark approach. DFlash predicts several tokens at once and has produced more than three times the acceptance of the previous best method in Baseten's production testing. Continuous retraining on live traffic can improve acceptance further, but requires storage, compute, and permission to use the data.
Local and data center inference optimize for different constraints
Kiely divides inference engineering into two practical settings. Local inference starts with the hardware available, then uses quantization, distillation, pruning, and model splitting to make the model fit. After that, engineers repair the quality problems caused by compression. Data center inference starts by getting a model running, then works on speed through techniques such as KV-aware routing, speculation, and disaggregation. The same optimization can have different value in each setting. A memory-saving method may help a machine with very limited local memory while adding too much computation for a production serving system.
Inference optimization increasingly includes a training step
Kiely says the old boundary between training and inference is becoming less clear. Many recent serving optimizations come from a dedicated training process rather than from changes made only at runtime. He describes a feedback cycle in which faster inference produces more data, that data trains a better model, and the improved model supports faster inference. This pattern appears in cache compaction and speculative decoding, where auxiliary representations or draft models are trained for a particular target model. His three practical areas for the talk are quantization, caching, and speculation.
TurboQuant saves cache memory but its decode cost limits data center use
TurboQuant uses polar coordinates to quantize the KV cache from eight bits to four bits. Kiely says this saves half the cache space and gives effectively double the bandwidth when moving the cache through system memory. The trade-off is extra computation during the forward pass, which cuts tokens per second by more than half. His team therefore continues to use NVFP4 mainly for weights and focuses on KV-aware routing, offloading, and sharing instead of compressing the cache with TurboQuant. He sees the method as more useful for local systems, especially long-context models running on hardware where memory is the main limit.
KV compaction learns a shorter memory representation
A KV cache grows with the amount of context, which becomes expensive at very long sequence lengths. Kiely describes a method from the Baseten research team called Still. It uses a learned bottleneck with a fixed set of query vectors. Those vectors cross-attend to the full KV cache and produce compact keys and values in one forward pass. The resulting memory is differentiable and short enough for the language model to attend to it as if it were the original context. Kiely contrasts this trained approach with earlier runtime compaction methods, including Attention Matching and Cartridges, which also seek high compression with little information loss.
Speculation moved from small draft models to trained drafters
Speculative decoding uses a draft model to propose tokens and the target model to verify them, allowing more than one token to be generated per forward pass without changing the final output. Early methods used a small model from the same family, but Kiely says these models were poor draft generators. Eagle 3 improved the setup by training a billion-parameter model on the target model's hidden states. That made it the strongest method in his comparison before DFlash. The main measure is acceptance rate, because rejected draft tokens reduce the speed benefit.
DFlash uses a diffusion language model to produce a sequence of draft tokens rather than generating one token at a time. It can be two or four times slower to run than a standard draft model, but it proposes eight or sixteen tokens in one window. Its tokens can attend to each other, which can raise acceptance. Kiely says a single DFlash forward pass is faster than the full Eagle draft phase while predicting more tokens, and reports more than a threefold improvement in production measurements against Eagle on a single B200 with Qwen 3 8B. The model uses bidirectional drafting inside a mask that preserves causal consistency.
DSpark was new research at the time of the talk, so Kiely does not provide production results. It combines a diffusion model with a sequential model. The proposed benefit is that the two drafting methods can work together to improve acceptance compared with using only an iterative diffusion speculator or only an autoregressive speculator. Kiely says his team had DFlash running in production, while DSpark was still under evaluation. The distinction matters because the reported gains for DFlash come from deployed measurements, whereas DSpark remained a research direction.
Continuous retraining can adapt speculation to live traffic
Speculative decoding depends heavily on the prompts and responses seen by a system. Kiely says continuously retraining the speculator on live traffic can improve token acceptance by 20 percent to as much as two times. This requires substantial storage and compute, permission to use the processed data, and a new speculator whenever the underlying model changes. He expects continuous speculation to become worthwhile for very large systems despite those costs. His forward-looking predictions also include stronger NVFP4 support, more disaggregation, and more system-wide movement of KV-cache data.
"The drawback is pretty big for TurboQuant. Turns out that you need to do additional computation in the forward pass to account for this during decode and it cuts TPS by more than half."07:11
Who should watch
You are serving language models in a data center and need to decide whether to spend effort on weights, KV-cache movement, cache compression, or speculative decoding.
You are building local inference systems where memory capacity matters more than the extra decode computation introduced by cache quantization.
You are evaluating trained draft models and want to understand the storage, compute, data-permission, and model-coupling costs behind continuous speculator retraining.
