A browser is useful for checkout, but using one for every stage makes an agent slow, expensive, and unreliable.
2
Agents must validate web content before sending it to a language model, because a successful HTTP response can still contain a CAPTCHA.
3
Search and scraping APIs can handle discovery and verification with smaller responses, explicit failures, parallel requests, and location-aware results.
Summary
Giedrius Šteimantas uses a personal shopping agent to explain the infrastructure layer that agentic systems need when they work on the open web. The original system used browser automation for discovery, product verification, and checkout. It sometimes returned CAPTCHAs, spent tokens processing invalid pages, and found products that disappeared at checkout because stock and availability depended on location. Šteimantas applies scraping practices to each stage. Discovery uses Oxylabs Fast Search API, which returns compact JSON and lets the agent choose URLs. Verification uses the Oxylabs Web Scraper API, which returns valid content or an explicit error, supports markdown, parallel requests, dynamic pages, and geolocation. Checkout remains browser-based because it requires interaction with a dynamic site. Playwright MCP connects to an Oxylabs headless browser with stealth, a residential proxy, and geolocation. His central advice is to use a browser only where needed and validate content before paying to send it to a model.
The shopping agent failed because browser automation handled every stage
Giedrius describes a personal shopping agent that first discussed a user's style, then gave a second agent prompts to find and buy products. Its creator used a browser automation framework throughout the workflow. The system was slow, expensive, and unreliable. Pages sometimes returned CAPTCHAs instead of product information. Šteimantas says the missing part was an infrastructure layer that would let the agent operate on the open web. He frames the problem as an implementation and web-access issue rather than a failure of the shopping idea itself.
Scraping infrastructure starts with cost control and content validation
Šteimantas gives two operating rules from his work at Oxylabs. Use a browser only when it is necessary, and validate the content returned by a request. An HTTP 200 response does not prove that the page contains the expected information. He also prefers lighter content because web pages include JavaScript, CSS, and HTML that may add bytes without adding useful information. These rules guide his redesign of the agent's discovery, decision, and purchase stages.
Browser-based discovery is slow, fixed, and vulnerable to blocks
The original discovery stage searched a predefined list of major retailers through a browser automation tool. Missing stealth led to CAPTCHAs and failed access, which required retries and made transaction costs hard to predict. The fixed retailer list also limited the products the agent could find. Heavy JavaScript made the process slower and more expensive. Location caused another problem: an item that appeared in stock during discovery could be unavailable during checkout because retailers showed stock, sizes, and options based on the user's location.
Search API discovery lets the agent choose URLs from compact results
For discovery, Šteimantas replaces the browser and fixed retailer list with Oxylabs Fast Search API. The API returns compact JSON in fewer than 2,000 tokens per response and averages less than 700 milliseconds. It gives the agent access to popular search engines and lets the agent formulate fan-out queries. The agent then selects relevant URLs from the results. Because the responses are small, the discovery stage can run quickly without complicated models.
A successful response can still waste most of the model's tokens
When the agent visits product pages for price, stock, descriptions, and details, browser requests can return CAPTCHAs or other blocked pages. Checking only response size and HTTP status can hide this failure. Šteimantas gives the example of opening ten websites, receiving valid content from only three, and sending all ten results to the language model. The model may identify the seven invalid pages, but the system has already spent tokens on them. In that example, 70 percent of the tokens are wasted. Compression does not solve the underlying problem because invalid content must be rejected first.
The decision stage works better when the scraper fails explicitly
Šteimantas rebuilds product verification with the Oxylabs Web Scraper API instead of running browsers directly. Valid content is returned, while CAPTCHAs and other blocks produce an explicit error, so the failed result is not sent to the language model. The API is a lightweight REST interface that can run hundreds of requests in parallel. It supports markdown instead of raw HTML, can render dynamic websites with a browser under the hood, and offers geolocation. Customers pay only for successful results, while failures are visible and have no charge.
Checkout is the stage where browser automation is necessary
The purchase stage needs a browser because it processes inputs on highly dynamic pages. Šteimantas and his friend both use Playwright MCP with a browser and a language model. The difference is the browser infrastructure behind it. Replacing the original browser with the Oxylabs headless browser is a drop-in change because it supports Playwright MCP. It adds stealth at the browser source-code level, a residential proxy, and geolocation. The agent can then select the prompted size, add the item to the cart, and complete the purchase with localized results.
The workflow should validate pages before spending model tokens
Šteimantas closes with three instructions for web agents. Apply scraping principles when building agent workflows. Use a browser only when the task requires one. Validate content before sending it to a language model. The missing infrastructure layer should handle web access so the agent can focus on the task itself. He returns to the cost rule at the end: cost matters.
"He was missing a layer. An infrastructural layer that would allow this agent to operate freely on the open web."02:05
Who should watch
You are building an agent that reads retail or other public web pages and currently use a browser for every request.
Your system treats HTTP status and response size as proof that a page is valid before sending it to a language model.
You need web results that can run in parallel, fail explicitly, or vary by user location before an agent makes a decision.
