Browser agents perform well on read tasks, but their success drops sharply when tasks require writing data or changing a website's state.
2
Infrastructure problems such as bot detection, authentication, email verification, and slow execution account for a large share of browser-agent failures.
3
Production systems should combine agents for dynamic tasks with deterministic workflows for repeatable steps, and should be tested with internal evaluations.
Summary
Jerry Wu and Wyatt Marshall define a browser agent as AI that controls a web browser to complete tasks for a user. They explain the observe, reason, and act loop, then use WebBench to measure agents across more than 5,000 read and write tasks on nearly 500 websites. Read tasks reach roughly 80% for leading systems, while write tasks perform much worse because they require longer trajectories, complicated forms, authentication, and interaction with stricter anti-bot systems. Failures also come from the surrounding browser infrastructure, including proxy blocks, CAPTCHA, and inaccessible email verification. Agents are slow because they repeatedly observe, plan, call tools, and retry. For engineers, the choice of read versus write use case should shape expectations and testing. The speakers recommend trying multiple infrastructure providers and combining browser agents with deterministic Playwright workflows. They expect progress in long-context memory, login and payment primitives, browser infrastructure, and model training environments.
A browser agent repeats an observe, reason, and act loop
Jerry Wu defines a browser agent as any AI that controls a web browser and executes tasks for a user. The agent observes the current browser context through a screenshot or extracted HTML and DOM. It then reasons about the next step, such as finding and clicking a search bar, and takes an action such as clicking, scrolling, or entering text. The resulting browser state becomes the next observation. This loop supports uses such as prospect-data extraction, software QA, job-application form filling, and attempts to automate traditional RPA workflows.
Evaluating browser agents requires realistic tasks and a controlled environment
Wyatt Marshall says evaluation is more involved than checking whether an agent completed a task. A useful dataset needs tasks that are realistic, feasible, specific to a domain, and scalable to create. Results can be checked with validation functions, human annotators, or an LLM judge. The infrastructure where the agent runs also affects its performance. WebBench separates read tasks, which mainly gather information, from write tasks, which change state on a site. The latter are harder both to build and to execute.
WebBench measures browser use across thousands of tasks and hundreds of sites
WebBench contains more than 5,000 tasks, with about half released as open source. The tasks cover read and write interactions across close to 500 websites and multiple categories. The benchmark is intended to provide a broad view of browser-agent behavior rather than rely on a simple flight-booking demo. Its leaderboard, technical report, GitHub repository, and Hugging Face dataset are available to the community.
Leading agents are already useful for information retrieval
On read tasks, OpenAI Operator with human supervision reaches about 80%, and the leading autonomous web agent performs around the same level. Several other agents come close. The speakers say agents are already good at browsing, finding information, and returning extracted data. The remaining failures often come from the internet or the execution environment. A sample task still includes difficult details such as several searches and filters, a Cloudflare popup, and multiple scrolling interactions.
Write tasks expose failures caused by long and difficult interactions
Performance on write tasks falls by about 50% or more for fully autonomous agents, while the supervised Operator baseline drops only around 10%. Write tasks take more steps, so each additional action creates another chance to fail. They also involve complex forms, data entry, dynamic interfaces, and authentication. Login requires managing credentials as well as handling the site's user experience. Anti-bot protections are often stricter when an agent tries to submit or change information. In one example, an agent could not submit a recipe after navigating a long flow with two login steps and dynamically added form fields.
Failures come from both agent ability and browser infrastructure
The speakers distinguish agent failures from infrastructure failures. An agent failure occurs when the site is available but the agent cannot close a popup, complete an interaction, or finish before a timeout. An infrastructure failure happens when the framework or browser setup prevents progress, such as a bot block, a proxy issue, or an email-verification step the agent cannot access. The boundary can be unclear, so WebBench sometimes groups the categories. Improving infrastructure could produce a large increase in overall success.
Latency makes browser agents unsuitable for many real-time uses
Browser agents are very slow. When an agent gets stuck, it may keep attempting actions until a timeout, which makes average execution time look worse. Even successful tasks take time because the agent repeatedly observes the page, plans, reasons, calls tools, navigates, and retries mistakes. Wyatt Marshall says this may be acceptable for an asynchronous, set-and-forget application. It is a serious problem for anything that needs to respond in real time.
Production systems should mix agents with deterministic workflows
For engineers choosing a use case, read systems such as research and large-scale information retrieval are already practical out of the box, though they still need testing. Write systems require much more rigorous evaluation before production release. The speakers also advise testing several browser-infrastructure providers because proxy access, CAPTCHA handling, and other behavior vary by site. At production scale, teams can use agents for long-tail, changing workflows and Playwright for stable steps that need predictable accuracy.
Login, payments, memory, and training environments are open areas for improvement
The speakers expect browser agents to improve through longer-term context, since write tasks can require far more steps than read tasks. They also point to browser primitives for login, authentication, and payments as major missing pieces. Better models will help with tool calls and write actions, while dedicated browser and computer-use training environments can provide examples for learning those behaviors. Their own company is working on training environments and sandboxes for this purpose.
"One of the big techniques that people utilize, especially for production scale use cases, is using a mix of browser agents for tasks or workflows that are more long tail, more dynamic or change often and mixing that with more deterministic workflows such as Playwright."17:12
Who should watch
You are deciding whether a browser agent is suitable for a product that reads information or changes data on websites.
Your agent works across login flows, forms, CAPTCHA, proxies, or email verification, and you need to separate model failures from infrastructure failures.
You are preparing a production browser-agent system and need a benchmark, provider comparisons, and a hybrid design with deterministic automation.
