AI agents can collect fresh web data when a question depends on current prices, events, reviews, or job listings.
2
A web-data pipeline must handle changing selectors, different page layouts, anti-bot systems, validation, deduplication, and delivery.
3
Yohan Raju's product agent combines Amazon product data and Reddit discussion, then uses Gemini to make a purchase recommendation.
Summary
Yohan Raju explains why an agent sometimes needs to collect information from the web at the moment a user asks a question. Static model knowledge or pre-indexed retrieval can answer stable questions, but current travel conditions, product prices, reviews, and job listings need fresh data. He first walks through the work involved in traditional scraping, including site-specific selectors, proxy infrastructure, browser behavior, validation, deduplication, and downstream delivery. Bright Data's MCP server gives an agent access to search, page extraction, site-specific data, and browser automation tools. A Glassdoor example shows the difference between a request that is blocked and one that reaches the page through the MCP setup. In the notebook exercise, Amazon data about a Stanley Quencher tumbler is combined with Reddit posts. Gemini analyzes price, rating, product details, stock signals, and sentiment, then returns a buy recommendation. The example is small, but the same pattern can apply to jobs and other changing datasets.
Traditional scraping becomes difficult as the number of sites grows
Raju says a scraper for one target site can take three to five days to develop when development, QA, testing, and the full pipeline are included. Ten sites already require substantial work, while hundreds or thousands create a maintenance problem. Developers must understand each site's structure and selectors, then keep existing crawlers working as new sites are added. If Amazon and two other sites change on the same day, the team must identify the changes, repair the scrapers, and determine whether customer SLAs and data quality were affected.
Changing selectors can break the rest of a data pipeline
A product page may expose a price and title through selectors that later change. When the price selector stops working, the scraper returns incomplete data and product analysis can be affected. Raju describes the usual response as a team repairing the scraper and backfilling data that was lost. An agent can automate some of this response by discovering the changed page structure and rebuilding the collection logic. The value is reduced downtime and less manual work when a site changes its layout.
Websites require different collection methods and block automated access
Amazon and Walmart may both contain prices, titles, and descriptions, but their layouts require different scrapers. Sites also use anti-bot systems and captchas to prevent collection. Raju describes a range of defenses, from sites that allow data-center IPs to sites protected by Cloudflare or Akamai that require residential IPs, a real browser, and human-like behavior. Clicking or scrolling too quickly, or blocking images, can trigger a block.
A traditional scraper has layers beyond the crawler itself
Raju's architecture starts with an actor such as a scheduler, cron job, configuration, or list of URLs. Requests then pass through proxy and anti-bot infrastructure, where the system may use data-center, ISP, or rotating residential IPs. The next layer parses and validates the response, checking fields such as price, reviews, and ratings. Post-processing handles deduplication, enrichment, batch processing, and delivery to cloud storage or internal systems. A failure in anti-bot handling or selector logic can force the process to restart and may require backfilling.
Agents choose between stored knowledge and fresh web collection
Raju separates questions that can use pre-trained or cached information from questions that need a live web request. A question about the best time to travel to San Francisco may use stored knowledge, while asking for the best time this year requires current events and conditions. He calls these synchronous and asynchronous paths. Current answers need a live collection setup, while stable or frequently repeated answers may be cached to avoid unnecessary web requests and token costs. Sites can also block direct access by bots such as ClaudeBot and GPTBot.
Bright Data's MCP server connects agents to web collection tools
Bright Data's Model Context Protocol server connects systems such as Claude Code or LangChain to more than 60 APIs. The tools include search engine results, page extraction as Markdown or HTML, site-specific collection for Amazon, Reddit, LinkedIn, and Instagram, and browser automation for actions such as clicking and scrolling. Raju gives an example where Claude Code searched for a person's social profiles, found the relevant pages, requested follower counts, and returned the results with context.
The MCP setup can determine whether a protected page is reachable
Raju shows the same browser prompt in Claude with and without Bright Data's MCP setup. The prompt asks the browser to get the contents of a Glassdoor URL. The MCP version reaches Glassdoor and returns results, while the other version is blocked. He connects this difference to Bright Data's proxy infrastructure and anti-bot tools, which can change the collection approach as a site detects and blocks repeated requests.
Fresh and historical job data are both needed for some models
Raju uses job data as an example of a dataset that cannot simply be collected once and forgotten. Openings, salaries, locations, and employers change, so a model for job patterns or resume matching needs continuous updates. It may also need data from previous months or years to understand changes in demand and salaries. The output has to remain structured and consistent, and downtime can affect the model pipeline and its results.
The notebook turns product data and online sentiment into a recommendation
The exercise uses a Bright Data API token and a Gemini API key. The notebook first retrieves product-level data from Amazon, including price, rating, sellers, features, images, and URLs. It then searches Reddit for discussion of the Stanley Quencher tumbler and passes the returned posts to an AI analysis step. The result describes positive sentiment alongside some mixed opinions, notes a discount and high demand with low stock, and answers whether the product should be bought.
"The sync path is, you know, if you really actually want to go out on the web and actually find what you're looking for and async is if you know it's okay to rely on pre-trained data."12:06
Who should watch
You are building an agent that needs current prices, reviews, job listings, or other changing web information.
Your scraping system breaks when sites change selectors or introduce anti-bot protections, and you want to understand the surrounding infrastructure.
You want a concrete example of combining structured product data with social sentiment before asking a model for a recommendation.
