GenAI applications fail because small changes in input syntax, meaning, or presentation can produce radically different outputs.
2
Static golden datasets miss the input-space corner cases that cause failures, and ordinary LLM judges have their own bias, calibration, and stability problems.
3
Haizing combines large-scale input simulation, automated judging, and search to find failures before deployment, including failures in text and voice applications.
Summary
Leonard Tang argues that the hard part of deploying GenAI is the last mile between a convincing demo and a system that can be trusted in production. He attributes much of the difficulty to brittleness: inputs that appear similar can lead to very different outputs. Static evaluation sets therefore provide limited coverage. Haize Labs' approach, which Tang calls hazing, sends generated stimuli to an application, scores the outputs, and uses those scores to guide further search for failures. The approach has two technical problems: building a judge that can turn subjective quality standards into reliable scores, and generating diverse inputs that expose corner cases. Tang describes agent-based judges, debate, self-verification, ensembles, reinforcement-learning-tuned judges, and optimization methods for searching natural-language inputs. He gives examples from bank loan applications, voice debt-collection agents, and a voice-agent evaluation suite.
The last mile is where a GenAI demo becomes difficult to trust
Tang says Haize Labs starts from the fact that AI systems are unreliable in practice and need pressure testing before release. A team can make a demo-ready product over a weekend, but getting it into production as an enterprise-grade system is much harder. He connects this gap to the unresolved problems of trust, reliability, and risk. His proposed answer is to run large-scale optimization and simulation before deployment, then use a battery of tests to check whether the system behaves as expected.
Brittleness matters more than ordinary nondeterminism
Tang distinguishes the problem from simple randomness. Teams often set temperature to zero and place models inside fairly deterministic workflows, so nondeterminism may not be the main operational problem. Brittleness appears when two ostensibly similar inputs differ slightly in syntax, semantics, or presentation and produce wildly different outputs. He gives examples involving Air Canada customer support, Character.AI, and a Chevrolet customer portal. These failures arise because GenAI applications are highly sensitive to changes in the input space.
Static evaluation sets provide too little coverage
The standard process collects a finite golden dataset with human-written expected outputs, runs the application on those inputs, and compares the results. Tang says this approach only measures performance on the chosen points. An application can achieve perfect results on unit tests and still fail on nearby inputs that the dataset does not cover. He also says quality is difficult to define: exact match, classifiers, LLM-as-a-judge, and semantic similarity each have problems when the real criterion depends on expert judgment.
An automated judge needs to be evaluated as carefully as the application
Tang describes judging as the task of translating subjective criteria into quantitative metrics. An off-the-shelf LLM judge can hallucinate, produce uncalibrated scores, and change its verdict when response order, context, or rubric wording changes. A score of five does not necessarily mean the same thing to a model and a human. Haize Labs therefore asks how to judge the judge before using it to guide improvements to the underlying application.
Agent-based judges spend extra compute on structured verification
One way to improve judging is to scale compute at inference time. Tang describes agent frameworks that use smaller language models in structured workflows, drawing on scalable oversight. Possible primitives include having models debate the stronger model's answer, self-verify a verdict, critique their own reasoning, and ensemble several judgments. He presents Verdict as Haize Labs' library for these patterns and says its systems beat several frontier models on an expert QA verification task while using a GPT-4o mini backbone.
RL-tuned judges can create criteria for each individual example
Tang's second judging approach is to train a model specifically for the evaluation task. He says reinforcement learning, including GRPO tuning, can address weak reasoning explanations and the lack of criteria tailored to an idiosyncratic dataset. He cites DeepSeek's SPCT, or self-principled critique tuning, which has a model propose data-point-specific criteria and critique the response against each one. In Haize Labs' experiment, 600-million- and 1.7-billion-parameter models reached competitive RewardBench performance through this kind of judge-time scaling.
Fuzzing searches for inputs that make the judge score the application badly
Tang separates ordinary fuzzing from adversarial testing. Fuzzing varies a customer happy path with reasonable in-distribution inputs. Adversarial testing tries to emulate prompt injection, jailbreaks, and other attempts to manipulate the system. Since natural-language input spaces are too large for brute force, Haize Labs treats the task as discrete optimization. It can use gradient-based token changes, tree search, Monte Carlo tree search, embedding-space search, and other methods to guide the search toward inputs that produce low judge scores.
The same search framework applies to regulated and voice applications
Tang describes testing a loan-calculation application for Hungary's largest bank against an 18-line code of conduct. Haize Labs found prompt injections, jailbreaks, and unexpected cases that the bank had not covered, allowing the team to patch the application before production. For a Fortune 500 bank's voice debt-collection agents, the search also varies audio conditions such as background noise, static, and frequency. Tang says work that took the bank's internal operations teams about three months took the platform about five minutes.
"What does bite you a lot when you're building AI apps though is when you send two ostensibly similar inputs to your AI application with maybe slight variance in the syntax or the semantics or the appearance of the text but all of a sudden you get wildly different outputs on the other side."03:17
Who should watch
You have an LLM application that performs well on a small golden dataset but still fails on unusual user inputs.
Your team needs to turn expert or subjective quality standards into an automated judge that can be tested and improved.
You are evaluating agents that operate across multiple turns, persistent conversations, or voice and audio inputs.
