AI models make fully deterministic CI impossible because small input changes can produce different outputs.
2
Zed treats evals as stochastic unit tests, then narrows failures into deterministic tests for specific parsing, editing, and protocol problems.
3
Teams can run repeated evaluations with pass thresholds while using trend analysis instead of treating every model variation as a build failure.
Summary
Nathan Sobo describes how Zed changed its testing approach after adding AI features to its Rust code editor. Zed previously worked hard to eliminate nondeterminism, including a simulated scheduler that replayed concurrent network interactions. Large language models changed that assumption because even a one-token input change can produce a different output. Sobo's answer is to treat evals as repeated, statistical tests, then use focused stochastic tests and ordinary deterministic tests to isolate failures. Zed runs some evaluations 100 or 200 times and can require every run to pass for a build gate. Other failures become tests for bounded model behavior, such as malformed XML tags, empty edit markers, indentation changes, arbitrary streaming chunks, and escaping. The practical lesson is that many AI failures still come from ordinary engineering problems around parsing and editing. Evals help find them, while conventional tests make the resulting fixes dependable.
Zed built its traditional CI around eliminating nondeterminism
Sobo says Zed would crash every eight seconds without its large test suite. Its tests can start a server, create two clients, and run many schedules of concurrent events through a simulated scheduler. When a particular interleaving fails, the team can replay and freeze it. This approach let Zed keep CI deterministic, even for systems involving networking and concurrency. The arrival of an LLM removed that control because changing one token in the input can produce a completely different output, even when sampling is tightly controlled.
A broad agent eval needs smaller tests to explain failures
Zed's first eval was data-driven: give the agent an input and inspect its output. The team then made the eval more programmatic. It compiles a headless copy of Zed, checks out a repository, runs the agent, and uses code assertions against what happened. This made it possible to isolate a simple failure in a tool that searched for a function and added an argument. The model saw an imprecise match, so Zed used tree-sitter to expand matches to syntactic boundaries. The stochastic eval exposed an algorithmic problem that could then be tested more directly.
Repeated runs turn a probabilistic behavior into a measurable test
For some evals, Zed runs the same scenario 200 times and sets a requirement that all 200 examples pass before the build can pass. Other tests run 100 iterations. This gives the team a way to measure behavior that cannot be judged from one model response. Sobo calls the resulting nondeterminism bounded when the system can handle the possible variations. The evaluation can still reveal a concrete defect, while the repeated runs show whether the defect is occasional or persistent.
Many AI failures are ordinary parsing and editing bugs
Zed added deterministic tests around several components that interact with model output. One parser test randomly chunks streamed input and checks that arbitrary boundaries are handled. A fuzzy-matching algorithm uses dynamic programming to tolerate approximate text matches. A streaming diff must distinguish deleted old text from old text that has not arrived yet. These components do not need model randomness to test them. They are conventional software problems that determine whether an AI editing feature works reliably.
The model sometimes emitted an empty old-text tag when inserting at the start or end of a document. A prompt change reduced the frequency, but the system still had to handle the remaining cases. XML tags were another example. Telling the model to close every tag correctly improved the result from about 40 percent of responses with a mismatch to about 95 percent without one. Zed still added a deterministic test for the remaining malformed output instead of assuming the prompt would eliminate it.
Indentation can be repaired after the model produces an otherwise valid edit
A model could identify the right Rust function and replacement while flattening its indentation. Zed handled this by computing the indentation difference between the buffer and the emitted text, then normalizing the edit. The team drove this behavior into a deterministic test with nested functions and an indented replacement. This let the editor accept a model response that had the right content but used a different indentation level.
Evals become useful when they narrow into ordinary tests
Sobo describes a progression from a broad eval, to a stochastic unit test focused on one part of the interaction, to a conventional deterministic test for the underlying behavior. Zed has implemented this in its existing test suite without external eval frameworks. The process keeps the model in the loop where its behavior matters, then removes randomness where the team can test a parser, matcher, diff, or edit transformation directly. Sobo's conclusion is that traditional software engineering skills still apply, with repeated statistical runs added for the parts controlled by the LLM.
"A lot of those problems are not advanced machine learning problems or anything. It's just stupid things that the model will try to do that we need to account for."13:13
Who should watch
You are adding an LLM to a product whose existing CI assumes that every test has a stable pass or fail result.
Your team needs to diagnose AI feature failures that involve parsing, streaming, text matching, or malformed tool output.
You want to add repeated model evaluations without adopting a separate eval framework or abandoning ordinary software tests.
