A browser controlled through Chrome DevTools Protocol can send input through the same internal path as a human browser user.
2
Agents should run a sense, act, verify loop and climb from synthetic actions to trusted input and human-like motion only when a page requires it.
3
Fast deterministic code should handle browser control, while an agent is called only for tasks that need visual interpretation, such as identifying tiles in a challenge grid.
Summary
Corey Gallon presents a method for making browser agents work with websites that resist automation. His premise is that a browser driven through Chrome DevTools Protocol can use the same input path as a human. The method combines a shell-based CLI, CDP browser control, and a repeated sense, act, verify loop. Gallon recommends starting with cheap synthetic actions, moving to trusted CDP input when a site rejects JavaScript events, and using human-like mouse movement or vision only when needed. His demonstrations cover Outlook email automation, trusted clicks on an add-to-cart button, Cloudflare Turnstile, image challenges, drag puzzles, and reCAPTCHA v2. The final design separates deterministic browser control from visual reasoning. Code handles clicks, screenshots, retries, and timing, while the agent identifies what appears in an image grid. Gallon is direct about the operational constraint: challenges expire, so putting a model in the middle of every interaction is too slow.
A CDP browser can send input through the same path as a human browser
Gallon's central premise is that a browser controlled through the Chrome DevTools Protocol can send clicks and keystrokes through Chrome's normal internal input path. He describes this as making the browser "just like a meat bag with a mouse," at least from the view of Google, Cloudflare, and other sites. CDP is the protocol behind the browser's developer tools, and agents can speak it directly. Gallon's Chrome Agent tool also lets an agent write code that replays CDP interactions. He groups the protocol's capabilities into digital senses: the DOM and accessibility tree provide structure and meaning, screenshots provide pixels, network traffic and logs provide signals, and input and navigation operate the page.
A CLI is faster and cheaper than putting an MCP server between every browser action
Gallon argues for shell-based tools because they can be programmed once and reused without a model in the loop. He cites an Arise AI study in which CLI and MCP approaches completed tasks successfully roughly 83 percent of the time, so he sees their capability as similar. The difference was in execution cost and speed. MCP took 71 round trips and eight minutes for a task that took a CLI seven turns and under a minute. Gallon also cites Anthropic's report that a CLI can be as much as 75 times cheaper in token cost. This matters most when a website challenge has a time limit.
The browser should be run through a sense, act, verify loop
The operating loop has three steps. The agent senses the page through the DOM, accessibility tree, screenshot, network, console, or logs. It then makes one move, such as clicking, typing, selecting, or navigating. It verifies the result through a channel different from the action. After a click, Gallon says not to ask the click whether it worked. Check the network or the screen instead. The loop repeats until the page reaches the intended state. When sensing, acting, and verifying fail to close the loop, that failure tells the agent to use a stronger interaction technique.
The ladder starts with synthetic actions and rises only when the page forces it
Gallon's three-rung ladder orders browser actions by how closely they resemble human input. Rung one uses the page's exposed APIs and synthetic JavaScript clicks. It is easy, free, and instant, so it is the default. Rung two uses real CDP input events through Chrome's input domain. These events travel through the same input path as a physical mouse and receive a trusted status. Rung three adds human input and behavior, such as screenshots, vision, a mouse path with dwell and jitter, and deliberate movement patterns. Gallon recommends exploring manually, climbing until the interaction works, then recording the successful path as code, an agent skill, or both.
A logged-in web UI can replace an API when enterprise permissions block API access
Gallon demonstrates rung one by sending personalized emails from Outlook's web client. A synthetic click opens the compose window, code fills in the fields, and another synthetic click sends the message. The sequence can then be reused for 20 emails or 200, while the agent generates personalized content. He explains why the web UI can be useful even when an official API exists. In a corporate Office 365 environment, API use requires app registration and admin approval, which an employee may not be able to obtain. A web login may already provide everything needed to perform the task, so the web UI becomes what Gallon calls a permissionless API.
Trusted browser events can succeed where JavaScript clicks are silently ignored
Gallon contrasts Outlook with an online store's add-to-cart button. The same JavaScript click that works in Outlook produces no error and no action on the store. He attributes this to Chrome's trusted or untrusted event status. The synthetic click is marked untrusted, and the page quietly drops it. Moving to rung two, the agent sends the click through Chrome's CDP input domain. Chrome marks that event trusted, and the item enters the cart. Gallon shows logs in which untrusted clicks fail while trusted clicks succeed. The cursor shown in the demonstration is added for visibility and does not move his physical mouse.
Visual and motion challenges require the agent to reproduce the relevant human behavior
For Cloudflare Turnstile, Gallon says the checkbox is hidden inside a closed shadow root and nested cross-origin iframe, so ordinary element lookup cannot reach it. His approach asks the browser for the iframe's screen position, calculates the checkbox location, and sends a trusted click to that position. For image text challenges, the agent takes a screenshot, reads the characters with vision, and types the answer using trusted keystrokes into the cross-origin widget. Drag puzzles add a motion requirement. The challenge samples the mouse trail, including jitter and changing speed. Gallon's agent identifies the gap with vision, eases into the drag, curves slightly, deliberately overshoots, and eases back.
Deterministic code should drive timed challenges while the agent handles visual interpretation
Gallon splits his reCAPTCHA v2 demonstration into a solver and an operator. The solver is deterministic code. It performs the trusted checkbox click, enters the challenge iframe, takes a screenshot of each grid, and rearms the challenge if a round expires. The operator is the agent, which looks at the fuzzy tile grid and identifies the requested object, such as a bus or bicycles. It returns the selected tiles to the solver and waits for the next round. Gallon says this division is necessary because the challenge runs on a clock and may contain several rounds. A model call on every click and visual step would consume the available time, while code can run at machine speed with one AI look per round.
The durable result comes from exploring a working path and recording it
Gallon says the challenge demonstrations are examples of a broader engineering method. First, give the agent a CLI so its behavior can be programmed. Then drive the browser through CDP and expose its digital senses. Run the sense, act, verify loop on the ladder, using the lowest rung that works. Explore until the interaction succeeds, then write the solution down as code, an agent skill, or both. That turns a one-time discovery into a reusable workflow. Gallon says the approach produced a repeatable solution for reCAPTCHA v2 and other challenges. He also says the demonstrations ran on infrastructure and accounts he owns, after noting that OpenAI had rescinded the warning on his account.
"You sense where you are. You make one move, you confirm that it landed, and then you iterate again."06:19
Who should watch
You are building agents that need to complete tasks inside websites where ordinary DOM automation sometimes fails.
Your team has browser workflows that need to run repeatedly without paying for a model call at every step.
You need to divide browser control between deterministic code and visual reasoning, especially when a site interaction has a time limit.
