An LLM gateway has to trade among availability, latency, guardrails, and cost during a degradation.
2
Per-request fallbacks, per-route P99 measurements, and explicit timeouts fit LLM behavior better than ordinary retry patterns.
3
Teams often need centralized governance rather than one centralized traffic gateway, so governance can be distributed through plugins and custom code.
Summary
Kanish Manuja explains the engineering choices behind production LLM gateways. A gateway sits between applications and model providers, handling routing, authentication, fallbacks, rate limits, and governance. During failures, teams cannot maximize availability, latency, guardrails, and cost at the same time. Manuja recommends per-request fallback instead of blindly retrying a slow and expensive provider, with careful capacity planning for the fallback provider. He argues that gateway-wide latency hides important differences between workloads, so teams should track P99 per model and route and set explicit timeouts. Reasoning models make this harder because the same prompt can vary from two seconds to 60 seconds. Guardrails need their own failure policy, timeout, fallback, and placement strategy. The gateway also creates a new dependency, so shared limits and bounded queues matter during retry storms. Finally, Manuja warns against deploying one gateway for an entire company when the actual need is centralized governance.
An LLM gateway forces trade-offs during degradation
Manuja defines an LLM gateway as middleware between applications and model providers. It handles routing, authentication, fallback, rate limits, and governance. At its center is a conflict among availability, latency, guardrails, and cost. When the system degrades, a team cannot maximize all four. The gateway should expose these choices to callers so they can select behavior for their use case. A single provider also sets the ceiling for availability: its outage becomes the application's outage.
Per-request fallback fits slow and expensive model calls
Ordinary retry patterns are a poor fit for LLM APIs. Exponential backoff and jitter consume the latency budget quickly, while blind retries multiply cost and tail latency. If another provider is healthy, a circuit breaker that simply stops calling the failing provider leaves useful capacity unused. Manuja recommends trying provider A and then provider B for the same request when A fails. Parallel requests can reduce latency, but they double cost. A provider that has been failing can still be removed from the load balancer and placed in cooldown before being tested again.
A gateway can keep provider failure counts in local instance memory or in shared infrastructure across the fleet. Shared counts support faster fleet-wide failover. Local counters avoid shared state, but deployment-size changes alter the meaning of the counters and the resulting behavior. This choice affects how quickly the gateway recognizes a failing provider and how consistently traffic moves away from it. It belongs in the design rather than being left to an implementation default.
Fallback providers need more capacity than primary providers
Fallbacks are not transparent even when providers support an OpenAI-compatible API. Tool-calling schemas, token limits, and stop reasons can differ, so cross-provider behavior needs testing and often a normalization layer. Streaming removes another control: once provider A has sent tokens, the gateway cannot switch to provider B or recall those tokens. The familiar "Something went wrong. Please try again." message can appear because the response has already committed to a provider. Manuja says the fallback provider should have more throughput, capacity, or headroom than the primary because it is the last line of defense.
A gateway can carry embeddings and classification requests that take less than a second, chat requests that take about three seconds, and reasoning requests that take much longer. An aggregate gateway latency number therefore hides outages in individual workloads. Manuja recommends tracking P99 per model per route. Timeouts should also be set per model class and route. Without a timeout, the gateway can treat a request as healthy while it is stuck. His comparison is that a reasoning model's normal latency can look like a chat model outage.
Reasoning models can produce very different response times for the same prompt. Manuja says he has seen production P99 latency rise to 60 seconds without a clear reason, while the same type of request can take two seconds. Router models add uncertainty because they choose which underlying model to run. He recommends fixing the reasoning level per route where possible and making requests as deterministic as the system allows. Another option is tail hedging: send another request after the primary consumes a chosen part of the latency budget, such as P90, to reduce the service's P99 tail.
Guardrails can block prompt injection, filter PII and toxicity, and enforce behavior requirements, but they are also dependencies that can fail. A system that fails open continues serving when guardrails are unavailable. A system that fails closed blocks the request. The right choice depends on the use case, and Manuja suggests choosing the worst case the team can live with. Guardrails should have time budgets so they do not determine the whole request latency. Teams can also use secondary providers, secondary checks, or cached decisions. Pre-hooks are safest but add serial latency, parallel checks fit structured outputs, and post-hooks suit monitoring and auditing.
A central gateway can create a company-wide failure point
The gateway itself becomes another dependency in the request path. Manuja recommends separating API keys by route and use case to limit noisy tenants. Gateways also need load shedding because retry storms are difficult to solve by simply adding servers. Internal queues should be bounded, and traffic prioritization can preserve important use cases under load. He questions whether companies asking for one central gateway actually need centralized traffic. Plugins and custom code can centralize cost tracking, rate-limit management, and other governance while traffic is handled by distributed gateway deployments.
"Your throughputs or your capacity or your headroom should be even higher for the second provider or the fallback provider because that's your last line of defense."06:00
Who should watch
You operate an LLM gateway and need to choose fallback, timeout, or circuit-breaking behavior for production failures.
Your gateway carries mixed workloads, including chat and reasoning requests, and aggregate latency metrics are hiding slow routes.
Your organization is considering one company-wide gateway but may only need shared governance such as cost tracking and rate limits.
