Agents That Own Their Inference

Du'An Lightfoot, Akamai, Khaja Omer, Akamai1:50:10 · Oct 2026 · 4,116 views
Thumbnail for Agents That Own Their Inference Watch on YouTube
TL;DR
  1. 1

    Owning inference makes sense when GPU usage is high and steady, data must stay private, and the team can support dedicated hardware.

  2. 2

    Agent workloads need measurements for time to first token, decode speed, throughput, KV-cache use, and the cost of repeated tool calls.

  3. 3

    Inference changes should be tested against the real workload because quantization and speculative decoding can improve speed, leave quality unchanged, or make performance worse.

Summary

Du'an Lightfoot and Khaja Omer explain how to decide whether an agent should use hosted APIs or dedicated GPUs, then walk through a hands-on Kubernetes and vLLM workshop. The agent loop repeatedly sends context to a model, performs tools, and sends results back, so latency and memory use accumulate across turns. The workshop measures prefill, decode, KV-cache capacity, streaming latency, throughput, and concurrency. It compares dense and mixture-of-experts models, then changes a Qwen deployment from BF16 to FP8 and checks both load-test results and simple output quality. Speculative decoding makes the central warning concrete: the draft model reduces performance from about 58 to 16 tokens per second because acceptance and model compatibility are poor. The presenters return to the baseline, tune serving flags, and raise the sequence limit. Their method is practical: establish a baseline, change one serving setting, rerun realistic tests, and evaluate output quality for the actual agent.

Key ideas
01:51

An agent's repeated model calls make inference part of the application design

Lightfoot describes an agent as a loop. A model receives a prompt, chooses an action, calls a Python tool or another program, and receives the result before updating its plan. Every loop generates more tokens. Tool calls, system prompts, context, and multiple turns therefore add both latency and memory pressure. The workshop focuses on taking control of that cost by running inference on hardware the team owns or rents directly. The point is not simply to run a model locally. It is to understand how the model behaves throughout the agent's full loop, including cases where a smaller model makes more tool calls or needs more reasoning tokens.

03:30

Dedicated GPUs fit steady, private workloads, while hosted APIs fit bursty demand

Lightfoot recommends considering dedicated GPUs when usage is high and steady, data must remain private, governance rules limit public providers, or the team has the hardware and expertise to operate inference. A GPU that sits idle is harder to justify. Hosted providers can make more sense when traffic is low or bursty, when the application needs a very capable model, or when the team cannot support self-hosted AI. He gives a workshop comparison using 200,000 tokens per day: the hosted bill is described as about $1,500 per month, while a rented GPU is about $1,000 per month. The actual model, GPU size, and workload still determine the result.

09:03

Serving measurements must separate single-request speed from aggregate throughput

The workshop uses vLLM because it provides continuous batching, paged attention, and an OpenAI-compatible API. Lightfoot distinguishes time to first token, decode or inter-token latency, and aggregate throughput. A single request reaches roughly 40 tokens per second, while many concurrent requests can produce higher total throughput even though each request still has its own generation limit. The presenters also track KV-cache utilization through metrics exposed to Prometheus and Grafana. For agents, observability needs to include each tool call, trace, span, token count, and latency step. That makes it possible to see whether a smaller model is spending its apparent savings on extra calls.

28:22

GPU memory is divided between model weights and the context held in the KV cache

Lightfoot explains that a 4-billion-parameter model in BF16 uses about 8 GB for weights, leaving the rest of the GPU for KV cache. Prefill reads the prompt and stores its information, while decode adds generated tokens and grows the cached context. Tool calls and system prompts consume that same budget. The notebooks use model configuration files to inspect layers, attention heads, vocabulary size, and other architecture details, then calculate how many tokens fit alongside the weights. This lets a team test whether a workload with many users and large contexts will fit or spill into slower CPU memory. Prefix caching can also reduce time to first token when requests share a prefix.

41:53

Dense and mixture-of-experts models have different memory and decode costs

A dense model uses the whole model to generate each token. A mixture-of-experts model can hold more total parameters while activating only a subset of experts for each token. Lightfoot uses this difference to explain why a larger mixture-of-experts model can decode faster than a smaller dense model in some situations. Decode is constrained by memory bandwidth, while prefill is more compute-bound. Reasoning adds another cost: a prompt that produces 35 tokens with thinking disabled can produce about 600 tokens when thinking is enabled. Agent workloads that need verification and several tool calls may require that extra reasoning budget, so model comparisons should include the actual agent behavior.

48:19

FP8 can free memory and improve throughput, but output quality still needs testing

Omer changes the Kubernetes deployment from the BF16 Qwen model to a pre-quantized FP8 version. Quantization reduces weight size, leaving more VRAM for KV cache and reducing memory movement during decode. In the workshop's load test, single-request speed rises from 37.5 to 53.4 tokens per second, and peak throughput at high concurrency also increases. Omer warns that quality can fail in workload-specific ways, including malformed JSON, broken function calls, hallucinations, or loops. He recommends evals for the actual application rather than assuming that a small quality change is acceptable. GPU support matters too. A format that is not hardware accelerated can make performance decline instead of improve.

01:05:09

Speculative decoding can slow an application when the drafter is incompatible

Speculative decoding uses a smaller draft model to propose several tokens, then asks the main model to verify them in parallel. The technique only helps when the acceptance rate is high enough to offset the work of running both models. In the workshop, a 0.6-billion-parameter drafter proposes four tokens for the main model. Performance falls from about 58 tokens per second to 16. Omer attributes the result to low acceptance, an unsuitable token count, GPU contention, and especially a draft model that was not trained correctly for the target model. He returns to the baseline and advises using a compatible drafter, often one released with the main model, while testing token counts and precision choices for the real workload.

01:26:38

The useful tuning point is where throughput flattens before latency becomes unacceptable

Omer treats concurrency as a curve to measure rather than a setting to maximize blindly. In the initial tests, throughput improves up to roughly 32 concurrent requests, then changes little while latency rises sharply at 64. The serving configuration includes knobs such as GPU memory utilization, maximum sequences, model length, and batch-token limits. The presenters change the maximum sequence setting from 32 to 128 and rerun the test. At 64 concurrency, throughput rises from about 800 to 1,100 tokens per second, while latency falls from about 2,000 milliseconds to about 800 milliseconds. The workshop's operating rule is to change one flag at a time, measure again, and stop when the hardware ceiling or the workload's latency limit is reached.

"You want to find the right flags for wherever the bottleneck is, find the bottleneck, figure out why that is happening, and what flag you can tweak to make sure that you're overcoming that bottleneck."Khaja Omer1:44:20
Who should watch
  • You are deciding whether an agent's traffic justifies dedicated GPUs or a hosted API, and need a way to compare cost, privacy, operational effort, and workload shape.
  • You run models on Kubernetes and need to connect GPU memory, KV-cache capacity, concurrency, batching, and serving flags to real latency and throughput.
  • You are considering FP8, speculative decoding, or another inference optimization and want a testing method that catches quality regressions and slowdowns.