AI failures can be mapped across technical strength, deployment controls, and business impact, with 13 sub-dimensions covering almost 98% of incidents.
2
Deployment controls, especially human review and monitoring, are more closely associated with harm than technical failures alone.
3
Evaluations should test the full deployed system through a failure-driven build-break-fix loop, then limit deployment to regions where the system is reliable.
Summary
Vishal Srivastava and Tanmay Sah studied 1,368 incidents from a public AI incident database and used them to build a framework for system-level evaluation. They group failures into technical strength, deployment controls, and business impact, divided into 13 sub-dimensions that cover almost 98% of incidents. A failure can involve several dimensions at once, but deployment controls such as human mediation, exposure control, and monitoring have the strongest association with harm. The speakers propose ranking AI use cases by materiality, then decomposing risk into failure frequency, whether the failure reaches a real decision, whether it is caught, and consequence severity. Evaluations for agentic systems should include retrieval, tools, permissions, human review, and recovery. They also argue that evaluators need evaluation because AI judges can miss failures. Their build-break-fix method starts with a system claim, tries to falsify it, adds controls, and repeats the test. An airline chatbot example shows how this could have caught incorrect refund advice before a customer relied on it.
AI failures need a structured framework because enterprise use cases outnumber review capacity
Vishal says AI failures have increased sharply during the foundational-model era, while enterprises have more AI use cases than they can review individually. He proposes examining each failure across three dimensions. Technical strength covers problems in data quality, model or system design, predictive validity, fairness, failure modes, and explainability. Deployment controls cover the authority given to the system, human mediation, exposure to users, and monitoring. Business impact covers financial consequences, legal consequences, and the number of affected users. The model is only one part of the system, so evaluation has to include how and where it is deployed.
Thirteen sub-dimensions cover almost all incidents and failures often involve several at once
The speakers report that their 13 sub-dimensions cover almost 98% of the AI failures they reviewed. Incidents can involve multiple dimensions, with an average of six failure sub-dimensions present in each incident. This gives teams a way to map what went wrong instead of treating every incident as an isolated event. The framework includes technical issues such as data quality and fairness, operational issues such as human review and monitoring, and consequences such as financial or legal impact. The speakers use this structure to decide what should receive attention first.
Deployment controls are more closely associated with harm than technical failures
The speakers compare how often each sub-dimension appears with its association to harm. Human mediation and harm detection through monitoring are strongly associated with harmful incidents. Scale and system design appear frequently, but their association with harm is lower in the analysis. When grouped together, deployment controls have the strongest association with harm, followed by business impact and technical failure. Vishal's conclusion is that a technically strong model does not provide enough assurance when the deployment setup lacks effective review or monitoring.
Materiality lets teams spend more evaluation effort on higher-risk use cases
Vishal proposes a materiality scale for ranking AI use cases by the harm associated with failure. Teams can then allocate more evaluation resources to use cases with greater financial, legal, or user impact. The scale can be specific to a company, and the level of risk can change over time. This approach avoids treating every use case as if it requires the same depth of investigation. The point is to decide where stronger controls and more extensive testing are needed before deployment.
System-level risk depends on failure, decision reach, detection, and severity
Tanmay decomposes risk into four parts: how often the model fails, whether the failure reaches a real decision, whether it goes uncaught, and how severe the consequences are. Deployment changes the resulting risk because the system may have different authority, human review, or monitoring in different settings. The same model can therefore create different risks depending on how it is used. An evaluation that measures only model behavior misses whether an error reaches a customer or causes harm before anyone detects it.
Agentic evaluations need to test tools, retrieval, permissions, review, and recovery
For an agentic system, Tanmay expands testing beyond model inputs and outputs. Teams should check whether retrieval returns the right information, what happens when a tool fails, whether human review catches mistakes, whether the agent stays within its permissions, and whether the system can recover. Traditional metrics such as accuracy and AUC ROC still depend on the quality of the data and assumptions behind them. Agentic systems add many possible inputs, tool responses, and action sequences, so selected scenarios must cover the deployed system rather than only a held-out model dataset.
The evaluator itself needs testing because AI judges can miss important failures
Tanmay says an AI agent or AI judge may generate tests or judge results, but that evaluator can make mistakes. Teams need to check whether the evaluator is reliable, what evidence it covers, and whether it applies to the system being deployed. Vishal calls the evaluator part of the governed system. This prevents teams from treating an automated judge as neutral evidence. A test can appear to pass because the evaluator did not represent an important failure or could not recognize it.
Failure-driven evaluation uses a build-break-fix loop to falsify system claims
Vishal connects this method to Peter Wason's work on confirmation bias and argues that evaluation should try to falsify a system claim rather than confirm it. Teams start with a claim about where the system can be trusted, actively try to break it, and then either restrict the system from the failing region or add supporting controls. They repeat the loop until they understand the reliable, uncertain, and high-risk regions. The result is a failure-region map that can guide deployment decisions for high-risk use cases.
The airline chatbot example shows how a narrow evaluation can miss harmful advice
Tanmay describes a customer who asked an airline chatbot about a discounted fare after a death in the family. The chatbot advised the customer to buy a regular ticket and claim the discount within 90 days. The airline later refused because its policy did not allow that request, and a tribunal held the airline responsible. A failure-driven evaluation would define the requirement that advice match approved policy, test questions about refunds after travel, add a policy check or human escalation when needed, and test again to see whether the control catches the error before it reaches a customer.
"The model alone or the technical strength of the model is just one part of the system. We have to look at the system holistically."Vishal Srivastava03:13
Who should watch
You are evaluating several AI use cases and need a way to decide which ones deserve deeper testing and stronger controls.
Your team tests a model in isolation, but the deployed system also uses tools, human review, permissions, or monitoring.
You rely on an AI judge or automated evaluator and need to test whether it can detect the failures that matter in your application.