# Computer-Use 2.0: Agents Just Got Multi-Cursor

Francesco Bonacci, Cua | AI Engineer World's Fair 2026 | 16:41

Source: https://www.youtube.com/watch?v=ZSQb5fzRFPw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/computer-use-2-0-agents-just-got-multi-cursor
Published: 2026-07-15
Tags: agents, benchmarks, computer-use, multi-agent, reinforcement-learning

## TL;DR
- Cua Driver lets multiple agents operate different applications through the operating system's accessibility layer without taking over the user's mouse or keyboard.
- CUABench evaluates computer-use agents with verifiable tasks and actively attacks environments for reward hacking before tasks enter the dataset.
- Replacing a standard computer tool with Cua Driver raised pass rate from 62% to 80% on a 4K benchmark while using 34% fewer tokens.

## Summary
Francesco Bonacci and his team describe a move beyond the usual computer-use loop, where an agent repeatedly reads a full screenshot and controls the foreground cursor. Cua Driver sends actions through accessibility APIs on macOS, Windows, and Linux, allowing agents to work in background windows while a person continues using the machine. The team built CUABench to test whether agents complete verifiable GUI tasks and whether the evaluations themselves can be trusted. Its electrical engineering tasks, built with Snorkel AI on professional circuit-design software, expose a sharp limit: the best tested agent fully passed six of 25 tasks, and every success involved editing an existing schematic. The talk also covers Cua Fleet, which keeps GPU workers busy by scaling a pool of sandbox environments on demand. The final Q&A touches on Android, where background tool use is more practical than full GUI control.

## Key ideas
### Computer-use agents started with a screenshot-and-cursor loop
[01:54](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=114s)
Bonacci describes the older computer-use pattern as a repeated human-like loop. The agent takes a screenshot, reasons about what it sees, plans the next step, and then clicks, types, or scrolls. The screen and hardware cursor are the main interface, so the agent typically takes control of the desktop while it works. He calls this "computer use 1.0." The Cua team has worked on GUI agents since its time at Microsoft, and the talk begins by using this older loop as the baseline for a different approach.

### Cua Driver lets agents operate in the background
[02:40](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=160s)
Cua Driver was released as an open-source project shortly before the talk. The team built its first version in a weekend after Codex released its computer-use model. Instead of taking over the visible screen, the driver connects agents to operating-system interfaces underneath the desktop. It supports macOS, Windows, and Linux, using accessibility trees or screenshots to understand a window. An agent can request window state, try an accessibility-tree action, and fall back to a pixel-level background click when needed. The driver handles differences between operating systems so the agent can work without disturbing the person using the computer.

### Accessibility APIs make multi-agent desktop control possible
[03:37](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=217s)
The driver uses undocumented APIs in the operating-system frameworks to send actions to windows that are not in the foreground. This lets an agent click or type without moving the user's hardware cursor. The demo shows agents working without taking control of the laptop, and the same general approach spans Apple platforms, Windows, and Linux. Cua maintains application harnesses to catch breakage across software releases. Early adopters including Clicky, Masqueno, and Droid Factory contributed upstream changes. The practical goal is to let several agents work on separate applications while the human keeps using the desktop.

### CUABench turns GUI tasks into verifiable evaluations
[06:34](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=394s)
Dylan explains CUABench as a way to test whether an agent uses its new computer-control abilities correctly. Each task has a setup function that creates the initial machine state, an oracle function that provides a reference GUI trajectory, and an evaluator that checks the resulting environment. The benchmark targets more than five desktop platforms and uses an SDK that can describe a GUI task in one Python file across those platforms. The dataset contains over 130 verifiable tasks across 42 environments and five platforms. Tasks can be reproduced through the command-line interface.

### Circuit-design tasks expose a severe limit in current agents
[08:20](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=500s)
The team built CUABench KiCad with Snorkel AI to test agents on electrical engineering work in professional circuit-design software. The evaluators simulate the circuits rather than checking only whether an agent clicked the expected controls. The best agent fully passed six of 25 tasks. Every one of those successes involved editing an existing schematic. When the task began with a blank schematic, every tested model had a 0% success rate. Across the models, no leaderboard entry exceeded 30% reward. The result is an uncomfortable measure of how far agents still have to go on open-ended GUI work.

### Cua Driver improves benchmark results while reducing tokens
[09:12](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=552s)
On the CUABench basic dataset at 4K resolution, agents using the built-in computer tool had about a 62% pass rate. Switching to Cua Driver raised that result to 80% and used 34% fewer tokens. Dylan attributes the difference mainly to the driver focusing on one window instead of processing the entire desktop. The evaluation therefore measures more than model behavior. It also shows how the interface given to the model affects both task success and the amount of visual or interaction data the model needs to consume.

### The benchmark tests its own environments for reward hacking
[09:37](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=577s)
CUABench does not simply assume that a successful-looking task is valid. Before an environment enters the dataset, a matrix of agents tries to break it and find reward-hacking paths. The team compiles those attempts into a code-review-style report, and only tasks that survive the process are admitted. Each recorded run can also be forked at any point in its trajectory. The team can then ask a model to predict the computer's reward, internal state, or another observation and compare the prediction with the forked state. Dylan describes this as making an agent's world model measurable.

### Cua Fleet keeps GPUs busy while sandboxes start and reset
[11:01](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=661s)
Rob focuses on a cost problem in reinforcement-learning training for computer-use agents. GPUs generate tokens, then often sit idle while a sandbox starts or resets. Large environments can make startup slow, especially when researchers need to pull a 40-gigabyte environment. Cua uses a pool of sandboxes and a demand-based autoscaler. It watches how many GPUs need environments and grows the pool on demand, even as that need changes during a multi-day training run. Extra sandbox capacity can cost less than leaving GPUs idle, so the infrastructure shifts startup cost away from the GPU workers and keeps them fully utilized. This approach provides instant sandboxes for Windows, Linux, Android, and macOS support is planned.

### Android supports background tool use more readily than full GUI control
[15:48](https://www.youtube.com/watch?v=ZSQb5fzRFPw&t=948s)
In the question period, Bonacci says Cua is speaking with the Arm team about an Android harness. Android can run containerized workloads, including an Ubuntu or GUI Docker container, which creates some options for background operation. He describes the Android ecosystem as more open to this kind of background computer use than iOS. The current direction is closer to tool use through the activity framework than to controlling a complete Android GUI in the background. The team can run work inside a container, but full multi-cursor-style interface control remains a different problem.

## Notable quotes
- "The trick here is really not having your agents take over your screen." (03:14)
- "The top agent that we tested only got a full pass on six out of 25 of these tasks." (08:40)
- "When we start the task from a blank schematic, the success rate drops to 0%." (08:44)
- "If you do RL training for computer use agents, you are probably leaving a lot of money on the table with idle GPUs." (11:01)
- "The answer is that it's evals all the way down." (09:59)

## Tools & references mentioned
- Cua Driver
- CUABench
- CUABench KiCad
- Cua Fleet
- Codex
- Snorkel AI
- Microsoft
- Clicky
- Masqueno
- Droid Factory
- Arm
- UI Automation
- AT-SPI
- AX
- Windows
- Linux
- macOS
- Android
- iOS

## Who should watch
- You are building agents that need to share a desktop with a person or with other agents, and foreground cursor control is getting in the way.
- You need GUI benchmarks with environment checks, reproducible tasks, and evaluators that inspect the resulting state.
- Your reinforcement-learning setup spends substantial time waiting for desktop sandboxes to start or reset while GPUs generate no useful tokens.