# Computer-use models will agentify the web, not APIs Dhruv Batra, Yutori | AI Engineer World's Fair 2026 | 21:00 Source: https://www.youtube.com/watch?v=Ki980nV0__0 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/computer-use-models-will-agentify-the-web-not-apis Published: 2026-08-14 Tags: agents, computer-use, cost, multimodal ## TL;DR - Computer-use agents will reach the long tail of the web through screenshots and interaction, because most sites will not expose APIs. - HTML is often insufficient because web state is calculated and rendered after the page loads, making pixels the practical source of truth. - Computer-use models are becoming accurate and cheap enough to run at scale, even though browser interaction remains slower than direct API calls. ## Summary Dhruv Batra accepts that agents will perform much of the action on the web, but rejects the idea that the web will quickly provide APIs for those actions. Large sites may add APIs, while the long tail of roughly 200 million active websites will keep its old infrastructure and slow-changing workflows. Batra shows restaurant menus hidden in PDFs and image galleries, school procurement information buried in scanned documents, and FOIA responses delivered through Google Drive. Reading HTML does not solve the problem because browsers fetch state asynchronously and render meaning from values such as quantity zero. Pixels are therefore the practical source of truth. Yutori's Navigator uses screenshots and clicks, with the ability to write JavaScript when that is faster, then checks the result visually. Batra reports 97% human-evaluated accuracy on a benchmark and describes falling latency and task costs. His forecast is an extra software layer that runs many browsers and returns structured results, effectively creating an API on top of the existing web. ## Key ideas ### The long tail of the web will not suddenly expose APIs [00:01](https://www.youtube.com/watch?v=Ki980nV0__0&t=1s) Batra agrees that agents may become the main actors on the web, but he rejects the usual conclusion that websites will meet them with APIs. The largest sites may provide APIs, while the long tail will not. He points to websites run by institutions with old infrastructure, including offices that still fax information to one another. Even if coding agents can generate endpoints, engineers will not receive unrestricted access to all of those institutions. Batra calls the idea that decades of infrastructure will be replaced in two, five, or ten years a fantasy. ### Restaurant menus expose the gap between an API-friendly example and the real web [03:23](https://www.youtube.com/watch?v=Ki980nV0__0&t=203s) A request such as finding gluten-free items on a restaurant menu rarely has the clean endpoint people imagine. Batra shows an easy site with visible text and prices, a medium site where the menu is hidden in a PDF, and a hard site where menu pages are JPEGs inside a gallery. The last case contains no usable text and can already challenge OCR. A system would need to find the gallery, open the images, interpret them, and extract the menu information. The page owner is unlikely to add an endpoint for a query such as 'give me your gluten-free items.' ### Public-sector procurement information can require a FOIA request [05:31](https://www.youtube.com/watch?v=Ki980nV0__0&t=331s) Batra uses school-district procurement as an enterprise example. A seller might want to know whether one of roughly 15,000 to 20,000 US school districts is currently procuring laptops. The information may appear in a finance or purchasing portal, a scanned solicitation PDF, or nowhere public. In his hardest example, the requester must file a Freedom of Information Act request. The office scans the email, places the scan on Google Drive, and attaches PDFs related to the request. Batra uses this workflow to question whether such an institution will publish an MCP server or a structured procurement endpoint. ### HTML cannot reliably recover state that the browser renders later [08:42](https://www.youtube.com/watch?v=Ki980nV0__0&t=522s) Reading the source does not always reveal what a person sees. On an NBA page, the initial HTML contains an empty placeholder for a game's score. The browser later makes an asynchronous call, receives JSON, and renders the score. A system that reads only the initial HTML misses the answer. Product availability has a similar problem. A product page may not contain the words 'sold out' or 'available.' Instead, a JSON response contains a quantity, and a script grays out and disables an option when that quantity is zero. ### Pixels are the source of truth for a web built for human eyes [11:33](https://www.youtube.com/watch?v=Ki980nV0__0&t=693s) Batra describes the browser as a rendering engine, similar to a game engine. What appears on screen may be calculated from data and scripts rather than written as plain text anywhere. Reconstructing the pixels from source code requires an exact inversion of the rendering process. Since websites were built for human consumers, Batra argues that machines need vision to operate across the long tail. Site-specific scaffolding does not generalize well. The general solution is the interface the web already provides to people: pixels in, actions out. ### Navigator combines visual interaction with code when code is faster [12:53](https://www.youtube.com/watch?v=Ki980nV0__0&t=773s) Yutori's Navigator began as a screenshot-in, button-click-and-scroll-out model. Batra gives the example of checking whether a discount code works under constraints such as a product condition, dates, or a minimum cart threshold. The agent visits the site, finds the product, adds it to the cart, applies the code, and checks the result. A later version can write JavaScript on demand, such as filling several form fields at once. It still checks the screen afterward, because the rendered result remains the source of truth. Multiple Navigator instances can also run in parallel in cloud sandboxes. ### Computer-use models are improving on both accuracy and operating cost [15:51](https://www.youtube.com/watch?v=Ki980nV0__0&t=951s) Batra says the progress he sees is faster than the online discussion suggests. On the Mind2Web benchmark, he reports that Navigator N1.5 reached 97% human evaluation, with eight incorrect trajectories out of 300, across tasks involving roughly 30 to 50 interaction steps. He says the benchmark should now be replaced with something harder. He accepts that visual browser use can be slow and expensive, then points to optimization. On the browser-use benchmarks he discusses, a smaller model has similar accuracy to larger frontier models while reducing latency and cost. For tasks requiring 20 to 30 steps, he gives costs of about 80 cents versus $2 to $3. ### A browser swarm can become an API-like layer above the existing web [18:28](https://www.youtube.com/watch?v=Ki980nV0__0&t=1108s) Batra's forecast adds another layer of software on top of the messy web. A user issues a task in natural language or with parameters. Behind the scenes, hundreds of browsers pretend to be human users, click through sites, and return a structured result. Falling costs, lower latency, and improving accuracy make this approach increasingly practical at scale. In some cases, Batra expects the work to run in a browser, take less than 100 milliseconds, and cost less than a penny. At that point, users may simply call the result an API even though it is powered by computer-use agents. ## Notable quotes - "My claim today and argument and what I hope to convince you today is that this last bit is wrong." (01:18) - "Fundamentally, what is happening here is this information that you are seeing on screen is not written somewhere as pure text." (11:27) - "The browser is a rendering engine." (11:50) - "Pixels are the source of the truth because the consumers of the websites are humans." (12:14) - "At some point you will say, 'Yeah, that's an API. Like why do I care?'" (20:23) ## Tools & references mentioned - Yutori - Navigator - Navigator N1.5 - MCP - Web MCP - Mind2Web - Google Flights - NBA.com - Google Drive - ChatGPT ## Who should watch - You are building browser agents and need to decide whether to invest in DOM extraction, site-specific scaffolds, or visual interaction. - Your product depends on data from small businesses, public institutions, or other websites that are unlikely to publish clean APIs. - You care about the operating cost of multi-step computer-use tasks and want Batra's view on model size, latency, and parallel browser execution.