# Your Self-Hosted LLM Needs to Say No

Harshul Jain, Audible & Parv Khatri, SAP | AI Engineer | 52:08

Source: https://www.youtube.com/watch?v=ntVpbyDTYaQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-self-hosted-llm-needs-to-say-no
Published: 2026-11-09
Tags: agents, benchmarks, coding-agents, inference, reliability

## TL;DR
- Admission control must decide whether to admit, defer, or reject a request before it takes GPU memory.
- Counting requests is a poor estimate of inference cost because requests consume different amounts of KV cache and may generate answers of different lengths.
- Admission policies should be benchmarked with realistic agent traces and scored by goodput under admission, which includes rejected requests in the denominator.

## Summary
Harshul Jain and Parv Khatri explain why self-hosted LLM systems need an admission layer in front of the inference engine. Coding agents can turn one task into dozens of concurrent calls, including retries and tool calls. The model does not know the length of its answer in advance, so every accepted request can hold growing KV-cache memory until it finishes or the client disconnects. The speakers propose three outcomes: admit immediately, defer with a retry time, or reject with an error. A useful policy needs information from both sides of the system, including request size, tenant, deadline, queue age, cache pressure, and replica load. They compare threshold rules, dual gates, optimizers, and formal deadline bounds, while noting that published systems do not cleanly fit coding-agent workloads. Their benchmarking guidance uses replayed agent traces, repeated runs, and goodput under admission. They also introduce admit perf, a tool for deploying and comparing policies against real engines.

## Key ideas
### An accepted request commits GPU memory for its whole lifetime
[02:13](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=133s)
Inference has a prefill phase, where the engine reads the prompt and builds the KV cache, followed by decode, where it writes the answer one token at a time. The cache grows with every generated token and stays reserved until the request finishes or the client disconnects. Harshul Jain stresses that the model does not know the final answer length when the request arrives. A short request and a long request therefore look similar at admission time, while their eventual memory costs differ. If memory fills, the engine may evict, swap, or error, and each option can cause delays and missed service objectives.

### Coding agents turn ordinary traffic into bursts of dependent calls
[12:30](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=750s)
The speakers contrast a person sending one prompt at a time with a coding agent that issues tool calls, retries, pauses, and undo calls. Asking an agent to fix one failing test can lead it to read many files, run tests, and make roughly 40 or 50 calls to the LLM. Some calls are short and some run for much longer. If a system admits requests independently, a task can end up with only part of its calls admitted, leaving the agent unable to finish. Admission control therefore has to account for the structure of agent sessions and the bursty traffic they create.

### Admission belongs before the inference engine
[08:55](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=535s)
The proposed pipeline is client, admission layer, queue or router, and inference engine. The agent knows when to retry, the gateway knows request counts, the router knows replica health, and the engine knows its current batch and KV cache. None of these existing layers has both the request details and the live engine state before the engine commits memory. The speakers focus on an ingress policy because it can work with an engine as shipped. Moving the decision into queue scheduling or decoding would require modifying and continually maintaining an engine such as vLLM or SGLang.

### A policy should return admit, defer, or reject
[18:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1129s)
At ingress, the policy can admit a request immediately, defer it with a time to retry, or reject it with an explanation. Before admission, the request uses no GPU memory. A defer response can use HTTP 503 and a Retry-After header when capacity should return soon. The client should sleep for that interval plus random jitter, so many agents do not retry at once. A 429 means the client or session is using more than its share. A 529 means the provider pool is overloaded. The speakers recommend reducing client concurrency for 429 responses and waiting, changing provider, or parking the task for 529 responses.

### Good admission decisions need request and engine signals
[29:49](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1789s)
The request side can provide input length, client or tenant, agent session, and deadline. The engine side can provide KV-cache use, the oldest queue wait, prompt-cache reuse, and replica-pool load. The speakers turn these into rules such as rejecting an input that cannot fit the memory headroom, deferring a session with too many calls in flight, rejecting a request that will miss its deadline, or admitting a cache hit because it is cheaper to serve. They advise using signals that change quickly, such as queue age, because a one-minute average can miss a burst of 40 calls in two seconds.

### Request counting does not measure inference cost
[27:48](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=1668s)
A fixed gateway allowance can keep clients or tenants fair, but it does not respond to GPU load. Counting in-flight requests per replica also treats a quick call and a long planning call as equal, even though their KV-cache costs differ. The speakers argue that a policy should estimate the actual cost of each request from live engine state, including KV memory and token use. They break policy design into four choices: the unit being admitted, the signal being watched, the rule that converts the signal into a decision, and the action taken. The unit may be a request, an agent session, or a tenant.

### Published policies do not directly fit coding-agent traffic
[34:29](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2069s)
The talk compares threshold policies, dual gates, optimizers, and formal bounds. The examples optimize different goals, use different engines, or run under offline workloads. Some admit individual requests, while a coding agent needs a policy that can reason about whole sessions. The speakers therefore do not present the papers as a leaderboard. Their decision tree starts with deadlines. If deadlines exist, use a deadline bound. Without deadlines, check whether the KV cache fills at peak load. If it does, use cache-related signals such as cache reuse and KV pressure. If it does not, queue depth may be the more useful signal.

### Goodput must count requests that the policy rejects
[40:22](https://www.youtube.com/watch?v=ntVpbyDTYaQ&t=2422s)
The proposed score is goodput under admission. Its numerator is the number of requests that meet the latency target, while its denominator is every request that arrived, including rejected requests. A policy that refuses 95 percent of traffic and serves the remaining 5 percent perfectly would have 100 percent SLO attainment if rejected requests were excluded. Under goodput, it scores 5 percent. The benchmark should replay realistic agent traces with fan-out, retries, and sessions, and it should sweep from half of measured capacity to twice capacity. A test that never exceeds capacity never forces the policy to say no.

## Notable quotes
- Harshul Jain: "A yes is not a moment. It's a commitment that lasts the request whole life." (10:40)
- Harshul Jain: "The short version is count is not a cost." (29:31)
- Parv Khatri: "A benchmark that never goes past capacity never actually tests the policy because the policy never has to say no." (42:22)
- Parv Khatri: "A policy that cannot see its signal should fail loudly and not just fail silently." (47:45)
- Harshul Jain: "When you are accepting the request in your inference clusters you need to have a mechanism to say no early." (50:47)

## Tools & references mentioned
- Audible
- SAP
- agentship
- LLM inference at scale handbook
- vLLM
- SGLang
- TensorRT
- OpenAI
- Anthropic
- Sonnet 4.5
- admit perf
- Qwen 2.5
- A10 GPU

## Who should watch
- You run vLLM, SGLang, or another self-hosted inference engine and coding-agent bursts are causing queueing, timeouts, or GPU memory pressure.
- Your gateway rate limit treats every request as equal even though prompts and generated answers have very different costs.
- You publish or evaluate inference policies and need a benchmark that includes rejected traffic and realistic agent retries.

## Editor's note

From the pack [Inference: serving models efficiently](https://aietalks.com/packs/inference):

These talks make inference tuning a comparison between configurations, not a hunt for one universal benchmark. Kitaru can replay the same recorded model workload against two runtime setups so a team can compare outputs alongside time to first token, completion latency and token use. The replay holds the application inputs steady; it does not reproduce GPU scheduling, queue pressure or cache placement from the original deployment, so those still need measurements from the serving system itself.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Deep dive on LLM Inference at Scale](https://aietalks.com/talks/deep-dive-on-llm-inference-at-scale) (Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher, 1:28:12)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [Hacking the Inference Pareto Frontier](https://aietalks.com/talks/hacking-the-inference-pareto-frontier) (Kyle Kranen, NVIDIA, 20:25)
- [Agents That Own Their Inference](https://aietalks.com/talks/agents-that-own-their-inference) (Du'An Lightfoot & Khaja Omer, Akamai, 1:50:10)
