Silent inference bugs can produce confident gibberish without crashes, warnings, or error logs.
2
The first bug happened when vLLM scheduled decode before prefill for a new Jamba request, causing Mamba to read stale state.
3
The second bug came from a 32-bit index wrapping after an offset passed four billion, and changing it to size_t fixed the overflow.
Summary
Asaf Gardin and Yuval Belfer describe two silent bugs they found while running Jamba, AI21's hybrid attention and Mamba model, in vLLM. The first produced gibberish roughly once per thousand requests. Lowering GPU memory utilization made it reproducible, and a logprob comparison against Hugging Face Transformers helped separate model behavior from inference-engine behavior. The real cause was a scheduler path that ran decode before prefill for a new request. Attention tolerated the ordering because it wrote KV data before reading it. Mamba read its state first, so it used stale data. The second bug appeared during RL training as a logprob spike every twelve steps. Increasing rollouts per prompt moved the failure to the first step, while reducing memory hid it. A 32-bit index had wrapped after passing four billion. The speakers recommend comparing logprobs with a baseline, constraining memory, changing workload knobs, and threading request identity through opaque execution paths.
Silent gibberish is an engineering failure, not a model-quality problem
The speakers begin with a production failure that has no crash, warning, or error. The output is bad but carries high confidence, so normal quality evaluation may not identify the cause. They distinguish this from a research problem where a model needs better answers. Their examples came from AI21's Jamba model, a hybrid of transformer attention and Mamba state-space layers, while the team was training and serving it. The failure required work inside the inference system rather than model optimization.
The first bug appeared rarely and only under a real workload
The first case, called the one-in-a-thousand gibberish problem, was difficult to reproduce because it did not appear in the first few hundred requests. It was rare enough to slow debugging, but common enough that shipping the system was unsafe. It occurred only in vLLM and needed multiple requests or workload pressure. Sending isolated prompts and small batches produced normal output, so the team needed a way to make the failure arrive quickly and consistently.
Reducing vLLM's memory allocation made the first failure deterministic
Asaf Gardin changed vLLM's GPU memory utilization from 90 percent to 20 percent, then ran many requests at once. With temperature set to zero, request 8,854 repeatedly returned gibberish. The memory setting controls space for weights, activations, and the KV cache. This created a short feedback loop for debugging. The team then compared vLLM with Hugging Face Transformers, which provided a plainer implementation of the Mamba kernels.
Logprob comparison cleared the kernels and exposed a misleading false lead
The team generated responses and logprobs with vLLM, then passed the full prompt and generated sequence through a Transformers forward pass using prefill. They applied softmax to the logits and compared token-level logprob distributions. They inspected the CUDA Mamba prefill kernel, compared tensors before and after it, and ran NVIDIA Compute Sanitizer without finding a memory problem. Forcing all computation through prefill made the gibberish disappear, which initially suggested a decode-kernel bug. That conclusion was wrong.
The scheduler ran a new request through decode before prefill
The forward pass contained tensors and matrices without a request identity, so the team added the request ID to a forward context and propagated it into Mamba. A breakpoint on the failing request showed that the scheduler sent it through decode before prefill. The normal request order is prefill followed by decode. The fix was to mark requests whose tokens had not been computed as prefill when the scheduler first classified them. The kernels were correct, but they were called at the wrong time for the wrong request.
Mamba exposed stale state because it reads before it writes
The ordering bug affected Mamba because its decode path reads the state before computing over it. A new request could therefore use state left by earlier requests. Attention behaved differently because it wrote KV entries before reading them, so stale data was overwritten. This explains why the issue appeared in the hybrid model's Mamba path and why simply inspecting the kernel arithmetic did not reveal the defect.
Changing rollout count moved the second failure and improved the debugging signal
The second case appeared during RL training as a logprob spike every twelve steps, before any weight update. The speakers wanted a setting that changed when or where the failure appeared, rather than only changing its severity. They increased rollouts per prompt from eight to 16, 32, 64, and 128. At 128 rollouts, the spike appeared on step one instead of step twelve. This gave them a much faster reproduction loop.
A 32-bit index wrapped around inside the Mamba state cache
Reducing GPU memory from 0.9 to 0.2 made the second issue disappear, but this was the wrong lever. The Mamba kernels used a 32-bit unsigned index. When the offset passed roughly four billion, the index wrapped instead of raising an error. With less memory, vLLM allocated a smaller state buffer, and the cache index never reached the affected location. Changing the variable from uint32 to size_t provided a 64-bit value on the relevant modern architectures and removed the overflow.
Both bugs required logprob forensics and attention to state-cache behavior
The speakers describe the cases as two scenes with one criminal. Both produced silent failures around the Mamba state cache. Memory pressure exposed both, although changing memory hid the second one. Logprob comparisons helped locate both problems. Their practical advice is to build a comparison script against another inference implementation, constrain memory to reproduce failures, change workload settings to study timing and location, and add identity to code paths where execution otherwise contains only anonymous tensors.
