An agent can inspect a website through Bright Data's MCP, generate a scraper, run it, and repair it when validation finds missing data.
2
Reusable scrapers reduce token use compared with sending every page through an LLM, with Rafael showing a 62% saving in one demo.
3
The system supports public-web collection and browser actions, but it does not support logging in and users still need to follow site terms and conditions.
Summary
Rafael Levi argues that the expensive part of web data work is maintaining scrapers, especially on sites with changing selectors, React interfaces, or strong anti-bot systems. His proposed workflow gives an agent access to Bright Data's MCP, skills, APIs, and remote browsers. The agent can inspect a site, identify selectors, generate a scraper with a defined schema, execute it, and check the output on a schedule. If a field disappears, the agent can repair the scraper. In a live example, the agent searched marketplaces for headphones and produced structured product data. Rafael says the generated scraper saved about 62% of tokens in one run, and that later script executions use far fewer tokens than asking an LLM to parse each page. The talk also covers Bright Data's markdown extraction, domain APIs, CAPTCHA handling, geo-targeted browsers, and browser actions such as filling forms. Rafael limits the workflow to public data and warns viewers to check website terms.
The agent builds a scraper instead of parsing every page
Rafael frames the problem around scale. Asking an LLM to parse 10,000 product pages would consume too many tokens, so the agent should first build a scraper and then reuse it. Bright Data's skills teach the agent how to create the pipeline, while its MCP lets the agent inspect page HTML and find the selectors needed for extraction. The result is a repeatable collection job rather than a one-off answer to a prompt. Rafael says this approach applies even when the goal is a personal search, such as comparing headphones or monitoring property listings.
A scheduled validation loop can repair broken collection jobs
Rafael describes running daily collections with an LLM worker starting every 30 minutes. The worker checks the collected data against a validation rule, then shuts down when the result is fine. If a data point is missing, the agent fixes the scraper instead of waiting for a person to notice the failure. He says the repair can take about five minutes, which avoids waking someone during the night. The same pattern can monitor a product, an apartment listing, or another public site on a schedule.
Bright Data's skills and MCP supply the pieces needed to generate a scraper
For the demonstration, Rafael uses Claude Code and a Bright Data GitHub page containing scraper-building skills and practices. He asks the agent to create a Walmart scraper with a keyword and a maximum page count. The agent retrieves the relevant instructions, uses the connected MCP to obtain page content, identifies selectors, and produces a scraper. Rafael says the MCP can return either full HTML or a markdown version containing the page text. The markdown option removes HTML tags when the agent only needs readable content.
A generated script can cut token use before the data reaches the model
Rafael contrasts reusable extraction with sending every page and its HTML to an LLM. In the three-page example, he says building and using a script saved about a million tokens. A later demo reports a 62% token saving for 90 products, although Rafael says the site may have relatively structured HTML. He also shows that the generated script returns JSON with a defined schema. Once the data is structured, the agent can answer questions about it without repeatedly parsing the original pages.
The MCP combines web unlocking, markdown extraction, APIs, and browsers
Rafael describes the MCP as giving an agent access to 66 tools. One tool can send a request to a URL and return HTML after handling the site's required headers, cookies, and CAPTCHA flow. Another returns page text as markdown. Bright Data also provides about 500 prebuilt APIs for different domains, so the agent can request structured product data without creating a scraper for supported sites. Remote browsers cover cases that need interaction, including clicking, form filling, and sites that restrict access by geography.
The browser infrastructure is designed to resemble human interaction
For actions that cannot be handled by a direct request, Rafael says the agent can start a remote browser with an IP from a chosen country. The browser can click controls, fill forms, and submit them. He describes mouse movement and typing patterns that are slower and less perfectly regular than programmatic actions. In his account, this makes the browser look more like a human session to sites that track interaction. He says even Claude Haiku is sufficient for browsing when the browser behavior is masked this way.
The workflow is limited to public data and excludes login
When asked about authorized sites, Rafael says Bright Data's workflow only handles public data and cannot log in. He tells users to read a site's terms and conditions, especially where a site forbids scraping or requires acceptance before access. He discusses legal disputes involving public web data, including cases involving Meta and Elon Musk, and gives his view that publicly available information remains public. The practical boundary he states is clear: do not use the system for data behind a login, and check the rules for each site.
The same pipeline pattern supports small personal automations
Rafael presents scraping as useful outside large enterprise collections. He describes setting a listener for a private house in a chosen area and price range, then receiving a notification when a match appeared. He also says he is monitoring a restaurant for an opening and wants the system to book a spot automatically. The agent can perform actions such as clicking through a search flow when a URL cannot express the needed state. These examples use the same inspect, build, schedule, and act pattern as a larger data pipeline.
"If for this example, there's always set a validation for data, right? Let's say the data point is missing something. Your agent fix it."02:07
Who should watch
You are building scrapers for sites whose selectors change and need a way to detect and repair missing fields without manual intervention.
Your agent spends too many tokens reading raw HTML or markdown for repeated web-data jobs.
You need public-web monitoring or browser actions such as geo-targeted searches, form filling, or booking, and want to understand the limits around login and site terms.
