# The Missing Layer After Launch Raphael Kalandadze, Wandero AI | AI Engineer World's Fair 2026 | 19:33 Source: https://www.youtube.com/watch?v=kZsf_Sfm7RU Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/the-missing-layer-after-launch Published: 2026-07-05 Tags: agents, coding-agents, computer-use, evals, observability ## TL;DR - Launching an agent is the start of the work because production conversations reveal failures that demos and conventional tests miss. - Operating agents need several feedback loops, including log monitoring, session analysis, browser-based customer simulation, and PR review. - Humans remain at approval boundaries while agents inspect conversations, diagnose causes, write tests, and prepare code changes. ## Summary Raphael Kalandadze argues that launching an agent product is only the beginning. Agent failures often appear inside successful-looking conversations: a wrong price, a dropped constraint, a poor tool choice, or a task that finishes without helping the customer. Unit tests, rules, and simulated conversations cover only part of the problem because inputs and model trajectories vary. Wandero AI uses agents to operate its production system. A log-monitoring agent investigates recent trajectories, traces problems into the code, and opens pull requests. A separate review agent checks those changes for root causes, risks, and edge cases. A session analyzer scores every conversation and finds broader patterns across production. A computer-use agent checks the product from the customer's point of view. Kalandadze is open about the cost, token use, and human bottleneck. His proposed answer is a connected operating system, or meta harness, that closes the feedback loop after launch. ## Key ideas ### Launching an agent starts the feedback loop [00:00](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=0s) Kalandadze says most agent talks end when the product ships, while production is where the real work begins. Teams need to watch hundreds or thousands of conversations, understand system health, and find failures they did not anticipate. He calls this the missing layer after launch. The product can be built quickly with current models, but that speed does not provide control over what happens with real users. A tight feedback loop helps the team improve the product every day, and Kalandadze says it can matter as much as the product itself. ### Agent failures hide inside apparently successful work [01:30](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=90s) Agents have no fixed set of flows, and users can ask for an effectively endless range of tasks. The same input can follow different paths, while a small input change can create a different trajectory. A run can struggle, recover through a workaround, and finish without producing a dashboard alert. Kalandadze gives a travel example: an agent builds an itinerary, but chooses a different service and makes pricing mistakes. The flow technically succeeds, yet the customer is unhappy. Unit tests and rules catch only one slice because customers do not repeat one predefined conversation. ### Log monitoring becomes a reasoning task [05:58](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=358s) Logs are an obvious place to give an operating agent access, but reading them requires more than filtering text. The agent must decide whether an event is a real bug or noise, inspect a fresh trajectory, and distinguish a symptom from a root cause. Kalandadze describes a flow that connects traces and trajectories to the codebase. The agent diagnoses the problem and sends a pull request, sometimes with a Slack alert for a critical issue. The resulting PR can include a description, metadata, Mermaid or ASCII diagrams, and HTML artifacts that help a human understand the change quickly. ### A separate review agent challenges proposed fixes [07:02](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=422s) The diagnostic agent is eager to fix problems, so Wandero uses a separate review agent with fresh context. It examines the PR from another angle, runs focused tests, criticizes the proposed fix, and can request changes or close the PR. Its feedback includes a summary, diagrams, risks, edge cases, and an explanation of whether more changes are needed. Kalandadze says the PR and review agents produce far more pull requests than the three-person team can handle. Humans therefore remain a bottleneck, but the prepared descriptions and artifacts let them understand a problem in minutes. ### Fast diagnosis needs a wider health view [12:12](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=732s) The hourly log-monitoring loop is good at finding local problems, but it cannot explain the overall health of the system. Wandero's session analyzer examines every conversation with stateful agents and assigns health scores. It looks for patterns across sessions and records details such as tool calls, sub-agents, summaries, and logical problems. Its dashboard includes analyzed session counts, cost, average score, success rates, trends, and AI-generated insights. For each important pattern, it describes why the issue matters, its likely root cause, the number of affected sessions, and a suggested fix. ### Production analysis needs the customer's point of view [16:20](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=980s) Logs, code, and session data can miss problems in the user interface. Wandero uses a computer-use agent to open the browser, log in, send messages, inspect sessions, and check how the result looks to a customer. Kalandadze says a general browser agent is slow, so the team built a skill that knows its website and DOM. This makes the checks faster, although they still use many tokens. When the browser agent finds a problem, it needs access to trajectories, the database, and other context to work out what happened. ### The operating system is a connected meta harness [17:49](https://www.youtube.com/watch?v=kZsf_Sfm7RU&t=1069s) Kalandadze calls the connected system a meta harness. The model alone is insufficient. Agents need access to the relevant tools, metrics, database, UI, traces, and code so their answers depend on the actual problem. In the intended loop, the system watches itself, understands failures, improves the product, sends pull requests, and issues notifications. His recommendation is to close this loop before treating shipping as the main achievement. The internal operating system is what gives a team a view of production and a way to act on it. ## Notable quotes - "I think the shipping is the moment when the real work begins." (00:54) - "So, you don't know what your agent will do until it is in the production." (03:14) - "So actually, you'll find out that the operating agent itself is an agent problem." (06:41) - "The most is not the model alone, and you need to build the agent, or a system, or a harness around it." (18:26) - "If you want to build a production agent, you need to close the loop first." (18:47) ## Tools & references mentioned - Anthropic - Codex ## Who should watch - You have launched an agent and need to understand failures that do not appear as crashes or red alerts. - Your team spends too much time reviewing production conversations, diagnosing recurring issues, or preparing fixes by hand. - You are deciding where humans should approve changes and where operating agents can inspect, test, and draft work automatically.