Computer use is easier than completing real work across the seams between applications.
2
Coding agents became trusted because their output can be run, tested, and verified, while most knowledge work has no simple unit test.
3
Perception agents close the computer-use loop by perceiving rendered interfaces, planning, acting, and checking whether their actions worked.
Summary
Antje Barth argues that computer-use agents have learned to click, type, scroll, call APIs, and run workflows, but still struggle with end-to-end work such as onboarding a new employee across several company systems. The hard part is the unverified work between applications. Coding agents advanced faster because code can be run and tested. Design decisions, reports, and user flows are harder to check. Barth proposes perception agents that share context with people by reading the rendered screen, reacting while work is happening, and checking their own results. She introduces an open-source harness with annotation and verification tools. Annotation lets a person mark an element and describe a change directly on the screen. Verification checks visual rules and user flows, then produces a report. She also shows how meeting audio can provide context when there is no screen. The talk is an invitation to try the tools and help build the patterns in the open.
Computer use handles individual actions but misses the work between applications
Agents can now drive browsers and are beginning to drive desktop applications, but Barth says clicking was the easy part. She uses onboarding a new team member as an example: someone must create accounts, add the person to Slack, book introductions, and order laptops. Each step may be possible for an agent, yet the full process crosses five different systems and often has no single owner. The work breaks down at the seams between applications and between steps. Giving an agent every individual tool does not make it capable of completing the whole job.
Reliability is what turns agent capability into trust
Barth separates capabilities such as tool use and workflow chaining from the harder problem of reliability. She asks the audience to consider an agent that succeeds 60 or 80 percent of the time. That might sound useful until the failures include destructive actions. Her example is direct: if an agent deletes a database one time in four, people will stop using it. Agents need reliability in the nines before people can trust them with work. The point is practical: a system can perform many actions and still be unusable when its failures are unacceptable.
Coding agents advanced because their output can be verified
Barth traces coding agents from autocomplete to writing functions and opening pull requests. Developers once felt they had to read every generated line, but code is now produced too quickly for that approach. Coding made this jump because its output is verifiable. You can run the code, test it, and check whether it worked. That provides a path from capability to reliability and then to trust. Broader knowledge work does not have the same property. Questions such as whether a report landed, whether a design is on brand, or whether the result matches the intended meaning cannot usually be answered by a unit test.
Shared context lets people solve messy work together
Humans handle work across disconnected systems by figuring it out with another person. Barth describes joining a colleague on a Zoom call, looking at the same problem and systems, and discussing what to do. Much of that work is not directly verifiable, yet shared context helps people resolve it quickly. Both collaborators see the same screen, so the person explaining the task has less to describe. Barth says agents do not necessarily need a bigger brain for this situation. They need access to the same context as the person, including what is visible and changing on the screen.
Today's agents can see a screen and take actions, but they often move on after clicking instead of watching the result or recovering from a failed step. Barth compares the needed behavior with robotics: a robot perceives its surroundings, plans what to do, and acts. On a computer, perception means taking in the rendered screen rather than scraping the code behind a page. The agent needs to understand layout, state, and what just changed, then keep up in real time. Barth wants an agent that can react while a person is still working, offer suggestions without waiting for another prompt, and understand what the person means from the same visual context.
Rendered-screen perception closes the computer-use loop
Perception agents add the part that ordinary computer-use agents handle poorly: looking at the result and deciding whether the action worked. They can read the rendered screen and confirm their own output instead of firing off actions and hoping. They do not require an API or backend process because they operate from the interface a person sees. This matters for software that exposes no API. Barth also argues that pointing at an element can be a more precise signal than writing a long description. A person can mark a heading or section and tell the agent exactly what to change.
Annotation transfers visual intent directly to an agent
Barth presents the annotation component of the open-source perception-agent harness as a Chrome extension. A user can draw around a heading, select an element by hovering over it, and request a change such as making text red or doubling its font size. The tool captures the location, style, and other screen details, then creates a complete summary for the agent. This reduces the back-and-forth caused by describing a visual change in text and asking the agent to infer which part of the page was meant. The person and agent can work from the same captured screen context.
Verification checks both visual rules and user flows
The verification component lets a team describe design rules in a design MD file, including colors, components, and layout. The agent checks its work against those rules with a visual check, such as whether the result is on brand and uses the right layout. It also walks through user flows in the application, performing actions such as adding or deleting a task. Afterward it writes a report showing which tests passed and what failed. Barth's example makes the division of responsibility clear: the agent performs the repeated checking, while a person reviews the report and decides whether to fix a violation or change the design rules.
Perception can include meeting audio when there is no screen
Barth extends perception beyond visual computer interfaces. In a design meeting with her colleague Giovanni, both wore B devices that recorded a transcript of the discussion. The transcript was brought into the website workflow as a detailed summary of the meeting and its ideas. A person could click Apply to send those changes to the agent. In the demonstration, the agent changed the background to yellow, the heading to red, and an emoji, then started verification. The same design rules flagged anything outside the approved guidelines. This example shows how spoken context can feed an agent even when the collaborators are away from a screen.
"If your agent one in four times deletes a database, you will never touch that agent again."03:54
Who should watch
You are building computer-use agents that can perform isolated actions but fail when a workflow crosses several applications.
Your team needs a way to check agent-generated designs, interfaces, or user flows without manually repeating every test.
You want to provide an agent with visual or meeting context and are interested in trying an open-source annotation and verification harness.
