AI systems need infrastructure that connects models to fresh web data, live search, and external information.
2
Web data products often become useful through repeated changes in scope, architecture, and delivery requirements.
3
At high request rates, realistic testing and telemetry become difficult because the systems measuring the load also add to it.
Summary
Patricija Žemaitytė argues that AI systems depend on infrastructure that can collect and deliver fresh public web data. She explains this through three projects. A video API began with a two-week deadline and a requirement to handle at least 5 petabytes per month, then expanded into support for transcripts, subtitles, search, metadata, and channel information. A search API had to move from a four-second baseline to sub-second delivery, even though browsers were required and repeatedly caused blocking. The team reached 550 milliseconds on average by removing time from many parts of the system. A web unblocker then had to grow from 10,000 to 60,000 requests per second. Load testing stalled around 20,000 because realistic traffic and trustworthy telemetry were hard to produce. Her conclusion is that web infrastructure has to keep adapting as targets, layouts, detection systems, client needs, and workloads change.
Fresh web data requires an infrastructure layer around AI models
Patricija Žemaitytė starts with the infrastructure behind AI systems rather than with models. Oxylabs builds systems that extract public web data at scale and connect it to models, agents, and databases. Training still matters, but she says it cannot provide everything a useful system needs. Models increasingly require live search and current external information. Without access to fresh data, a model remains limited by its existing knowledge. Her product work began with teams handling unusual client requests, tight deadlines, and products that did not yet exist.
A video download request became a wider product family through repeated changes
A sales team returned from San Francisco with a request for a video API for AI training. The deadline was two weeks, and the client expected at least 5 petabytes per month. The work required collection, transfer, storage, and delivery, so it quickly looked like infrastructure rather than a simple downloader. After the first version shipped, the client asked for transcripts. Those requests actually needed subtitles, so the team changed the product. Search for videos in particular languages came next, followed by metadata and channel information. In about three months, the one-off request had become a video API suite.
A fast search API needed a redesign after a successful first version failed in reality
In 2024, a client asked for search results in less than a second, while the existing scraper averaged about four seconds. The team built a faster version, but the client did not test it and the market was not ready, so they shelved it. A later request required zero data retention, sub-second latency, support for geolocation and query parameters, and a testable solution within two weeks. The first new version reached about 650 milliseconds at P90. During the client call, however, the system was blocked badly. That forced a second iteration and exposed the difference between development, testing, and real-world operation.
Browsers forced the team to find latency in many small parts of the system
The second search implementation needed browsers because of the way the target systems behaved. Browsers were useful but slow, expensive, and complex, which conflicted with the sub-second requirement. The team therefore reviewed layouts, parsers, sessions, and proxies, looking for time to remove from each part. Žemaitytė says the result came from small decisions that added up rather than from one large breakthrough. The resulting fast search API delivers fresh data directly into AI workflows with an average latency of 550 milliseconds. Daily traffic also grew from 400 million requests to almost 6 billion, changing how the team had to think about costs, observability, and failure domains.
High request rates are end-to-end scraping workloads, not simple HTTP counts
A separate demand required the web unblocker to grow from about 10,000 requests per second to 60,000 in less than two months. Each request represented an end-to-end scraping job with routing, rendering, proxy handling, browser execution, parsing, retries, normalization, and delivery. Adding servers alone could not solve the problem. The architecture needed reliable central components, observability that remained trustworthy, and tests that resembled actual usage. The team hit a wall around 20,000 requests per second during load testing. The system was working, but they could not tell with confidence whether it could go further.
At scale, telemetry and realistic traffic become part of the engineering problem
Synthetic traffic was relatively easy to generate, but the team needed organic data that behaved enough like real client traffic to produce useful results. Metrics and logs became difficult to collect and process at high volume. The telemetry itself added load and complexity, so measuring the system affected the system being measured. The team scaled gradually and eventually accepted that production traffic would provide the real test. That test went well, but the project continued to expand. The internal name Project 60 already had become Project 150, with discussion of roughly 100,000 requests per second.
Web data infrastructure has to adapt continuously as external conditions change
Žemaitytė describes Oxylabs as more than a proxy provider. Its infrastructure reaches the open web, collects data, handles antibot systems and browsers, and delivers information into AI systems. The maintenance burden remains large because targets, layouts, detection systems, and client requirements change. She calls this an adapt-forever business rather than a build-once business. Her final claim is that the next generation of AI will depend on infrastructure connecting models to reality, pushing web data into pipelines, databases, agents, and tools, while the infrastructure provider handles the changing conditions underneath.
"This is really honest moment about when you think about infrastructure and systems, because this is a kind reminder that there is a difference between system that works in development, system that works in a test, and system that actually survives reality."09:49
Who should watch
You are building retrieval, agent, or AI training systems that depend on current web data and need to understand the infrastructure underneath them.
Your first implementation works in tests, but you are preparing for blocking, browser overhead, changing layouts, or other production conditions.
You need to load-test a scraping or data delivery system and are unsure whether your traffic model and telemetry reflect real usage.
