# Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work | AI Engineer Summit 2025 | 17:44 Source: https://www.youtube.com/watch?v=kjSGc7uwDo8 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/beyond-apis-how-ai-web-agents-are-automating-the-long-tail-of-knowledge-work Published: 2025-02-22 Tags: computer-use, data-pipelines, privacy, tool-use ## TL;DR - AI web agents can use natural-language instructions to act across browser pages and extract structured data into Google Sheets. - A browser extension can work with logged-in pages, background tabs, and local content without storing users' passwords in a cloud browser. - The presenters argue that text-based page processing makes multi-tab automation cheaper and less prone to hallucination than screenshot-based agents. ## Summary The presenters show rtrvr.ai as a Chrome extension that turns browser tasks into natural-language workflows. It can search LinkedIn, follow pages, extract article fields into Google Sheets, process PDF links from a search page, compare Amazon products, and change page settings before collecting data. It can also summarize selected documents, research companies and pricing pages, calculate fields such as revenue growth, call user-defined third-party functions, send WhatsApp messages, and generate graphs. The system works inside the user's browser, including logged-in and paywalled pages, and can process background tabs in parallel. The presenters contrast this with vision-based agents that rely on screenshots and cloud browsers. They argue that reading page text reduces hallucination and cost, while a local extension avoids password storage and provides more personalized results. Their longer-term idea is collaborative dataset construction from many people's browsers. ## Key ideas ### The browser is still a bottleneck for routine work [00:01](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=1s) The presenters say people spend hours copying information between websites, Google Sheets, and CRMs. Some companies outsource scraping, which they describe as expensive and unreliable. Others use RPA bots that break when a website changes. Data is also split between websites and APIs, making users combine information manually. rtrvr.ai is presented as a Chrome extension with an AI web agent that can act across pages and extract structured data into Sheets. ### Natural-language browser tasks can combine actions and extraction [01:29](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=89s) The LinkedIn demonstration uses a prompt to find and follow the Latent Space podcast page. The agent fills in the search field, interacts with page elements, and notices when the page is already being followed. The same system can extract fields from every article on a page into Google Sheets. The presenters also combine actions and extraction, so an agent can change a page state before collecting information. ### Parallel tabs turn research tasks into spreadsheet workflows [03:05](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=185s) The agent can take a Google Sheets column of URLs, open the pages, and process them as independent subtasks in new tabs. The demonstration extracts fields from the first five PDFs in an archive search. It can also compare Amazon product pages, infer useful fields when the prompt is empty, collect source image URLs, and select the most recent review from a dropdown before extracting the review details across several tabs. ### The agent can explore beyond the pages initially opened [07:05](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=425s) For market research, the presenters ask the system to examine agent companies and collect strategy, features, and pricing. The agent first generates a schema from the request, then moves from company homepages to pricing pages to find the requested information. Users can specify which websites or URLs the agent may visit. The presenters describe this as a deep-search capability that writes the resulting information into Google Sheets. ### Browser data can feed calculations and user-defined tools [08:44](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=524s) A stock research example starts with a Google Sheet containing companies. The agent is asked to collect P/E ratios from Yahoo Finance, retrieve two years of revenue, and calculate revenue growth as a new field. When given a company website instead of Yahoo Finance, it recognizes the mismatch and finds the requested information elsewhere. The presenters also demonstrate dynamic function calling, where users provide tool information so the agent can invoke an integration such as WhatsApp and send messages to phone numbers in a sheet. ### The extension handles documents, graphs, and communication tasks [06:25](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=385s) The presenters select design documents and ask for summaries and key points. The sources can include Google Docs, PDFs, and Google Sheets, and the agent reports what each document contains. A graph tool then generates a data analysis graph from a dataset. They argue that language models can both extract data and represent it in formats that users request. The same workflow can connect spreadsheet data to outside communication tools. ### Text-based processing is presented as cheaper and more parallel than screenshots [12:11](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=731s) The presenters compare rtrvr.ai with agents that use screenshots, including systems associated with Anthropic Claude, OpenAI, and Google Mariner. They argue that vision-based systems are more prone to hallucination and require many screenshots, which raises cost. Because rtrvr.ai reads page text directly, it can process background tabs that are not rendered and take actions across multiple tabs in parallel. ### A local browser extension avoids some cloud-browser limits [12:48](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=768s) The presenters say cloud browsers can show generic or different content from the user's browser and may require proxies to route network requests. rtrvr.ai runs as an extension in the user's browser, so it can access the same logged-in and local content the user sees, including subscribed pages. They say it does not store passwords. Long tasks are split into subtasks in separate tabs, which they claim reduces failures compared with one long action on a single tab. ### The long-term idea is collaborative dataset construction [16:24](https://www.youtube.com/watch?v=kjSGc7uwDo8&t=984s) The presenters describe a future in which people use the extension to contribute browser-based work to shared datasets. Their example is a collection of local government events in the San Francisco Bay Area, gathered from hundreds or thousands of websites. They say this kind of dataset is currently too expensive to build centrally, while distributed contributions from people's laptops could make it practical. ## Notable quotes - "Your browser is a bottleneck for most of your workflows right now." (00:01) - "We can say that our approach to solving this problem is very cost effective and it probably cost less than a penny to do this page extraction." (02:43) - "With rtrvr you don't have to share any of your passwords." (13:49) - "We are all in for productivity and automation use cases because we see that AI is perfect to automate these manual and repetitive tasks." (15:20) - "Our long-term goal is allowing people to collaborate across their own local laptops to collaboratively construct data sets." (16:46) ## Tools & references mentioned - rtrvr.ai - Google Chrome - Google Sheets - LinkedIn - Latent Space podcast - Google Docs - Yahoo Finance - WhatsApp - Instagram - Facebook - Anthropic Claude - OpenAI - Google Mariner - Browserbase - RPA ## Who should watch - You are building browser automation for internal operations and need to work with pages behind logins or subscriptions. - Your research workflow starts with URLs, documents, or spreadsheets and ends with structured data, calculations, or summaries. - You are comparing cloud-browser agents with an extension that runs in the user's own browser. ## Related talks - [AI Didn't Kill the Web, It Moved in!](https://aietalks.com/talks/ai-didnt-kill-the-web-it-moved-in) (Yohan Lasorsa, Microsoft & Olivier Leplus, AWS, 52:55) - [The Web Browser Is All You Need](https://aietalks.com/talks/the-web-browser-is-all-you-need) (Paul Klein IV, Browserbase, 17:31) - [Building AI Agents that Actually Automate Knowledge Work](https://aietalks.com/talks/building-ai-agents-that-actually-automate-knowledge-work) (Jerry Liu, LlamaIndex, 17:57) - [The Current State of Browser Agents](https://aietalks.com/talks/the-current-state-of-browser-agents) (Jerry Wu & Wyatt Marshall, Halluminate, 21:13) - [The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans](https://aietalks.com/talks/the-dark-arts-of-web-automation-teaching-agents-to-use-websites-like-humans) (Corey Gallon, Rexmore, 21:38)