# How agents broke app-level infrastructure

Evan Boyle, GenSX | AI Engineer World's Fair 2025 | 13:32

Source: https://www.youtube.com/watch?v=_-oIuRH4oGA
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-agents-broke-app-level-infrastructure
Published: 2025-06-03
Tags: deployment, observability, reliability, workflows

## TL;DR
- LLM applications take seconds, minutes, or hours, so infrastructure built for short web requests does not fit their workloads.
- Long-running agent workflows need resumable streams, background execution, retries, tracing, and APIs that survive page refreshes.
- Separating the API layer from the compute layer lets each scale independently and lets workflows continue without an open browser connection.

## Summary
Evan Boyle argues that infrastructure assumptions from web applications no longer fit products built with LLMs. A normal request might take tens of milliseconds, while an AI request can take seconds and an agent workflow can run for minutes or hours. Existing serverless platforms often time out, lack native streaming, and make resumability difficult. Boyle describes an infrastructure pattern built around reusable, independently testable components and workflows. The API layer starts work in a separate compute layer, then communicates through a Redis stream that stores status and output. This lets users refresh or leave a page without losing progress. The system also adds retries, tracing, heartbeats, error handling, and automatic REST APIs. Boyle advises teams to start with simple workflows while planning for longer execution, and warns that long-running workers make deployment and draining much harder.

## Key ideas
### LLM products turn short requests into long-running workflows
[00:02](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=2s)
Boyle describes a common progression from a prompt with a few tool calls to a workflow with chained prompts, evaluations, and controlled context. That can move execution from about 30 seconds to a couple of minutes. Teams then ingest large amounts of context from sources such as a user's inbox or GitHub, which requires many more LLM calls. The infrastructure problem changes as the workload grows. A web service that once fetched data and returned it in tens of milliseconds now has a P1 latency of a couple of seconds at best, while background workflows may run for minutes or hours.

### LLM dependencies make reliability and capacity harder
[02:02](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=122s)
AI applications depend on services that can be slow, unavailable, or rate-limited. Boyle describes outages affecting OpenAI and Gemini during the same period, so simply failing over to another model provider may not help. Batch processing documents or onboarding customers also creates bursty traffic. Teams without high rate limits may need to spend thousands of dollars to reach the capacity required for serious experiments. These conditions make the short-request assumptions of web infrastructure a poor fit for applications that depend on many model calls.

### Serverless defaults do not fit workflows that last minutes
[03:20](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=200s)
Boyle says most serverless providers time out after five minutes, while some also limit outgoing HTTP requests. None of them provide native streaming for these workflows. That forces teams to add streaming at the application layer. In one content-generation product, a workflow took three to four minutes and showed research, outlining, and section-writing steps. The team avoided making workflows longer than five minutes because moving away from its serverless provider required deep infrastructure changes and took much longer than expected.

### Users need progress that survives navigation and errors
[04:34](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=274s)
For onboarding, a product may scrape a website for several minutes and make hundreds of LLM calls. The user should be able to use the product immediately while ingestion continues, with the interface showing how the data is becoming richer. For content generation, the user may leave or refresh the page while the agent works. The workflow must preserve intermediate status and final output. Boyle also points out that an error after five minutes is more frustrating than an error on a simple form, so the system should resume from the previous point instead of forcing the user to start over.

### Reusable components make agent workflows easier to test and retry
[07:14](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=434s)
GenSX uses an infrastructure-aware component model inspired by React's component model, applied to the backend. Components are reusable, idempotent, independently testable steps. Workflows combine those components. One example fetches Hacker News, analyzes posts, generates reports, writes a tweet, and edits the report. Each component gets its own retry and error boundary, while traces cover both the overall workflow and the components inside it. The framework wraps the OpenAI SDK to add tools such as retries and tracing while keeping the familiar model interface.

### A Redis stream separates API traffic from background compute
[10:09](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=609s)
The proposed platform has separate API and compute layers. The API invokes a sandbox program in the compute layer and passes it a Redis stream ID. After that, status and output move through the Redis stream rather than through a persistent API-to-compute connection. The sandbox emits heartbeats so the system can detect crashed or dead workflows, restart them, or notify the user. Since the API reads the stream instead of reading directly from the compute process, the UI can recover the full history after a refresh or a broken browser connection.

### Long-running workers change deployment operations
[11:59](https://www.youtube.com/watch?v=_-oIuRH4oGA&t=719s)
Boyle advises teams to start simple because an hour-long workflow is unlikely to be the first version of a product. They should still plan for workflows to become longer. That means separating the compute and API planes and using Redis streams for resumability. Deployment needs special care when workers can run for 60 minutes. Teams must think about how to drain workers and use a blue-green deployment pattern without interrupting active workflows. Boyle is direct that this infrastructure is easy to start and difficult to get right.

## Notable quotes
- "The infrastructure that was designed for the last 10 years of the web is not really designed for the applications that we're building today." (01:44)
- "We want users to use the app immediately." (05:08)
- "You don't want to abstract. You want to share, you want to compose, you want to reuse code." (07:48)
- "None of the work is lost if the new user navigates away or if the web browser terminates the connection to the API server." (12:00)
- "Easy to get started, but very hard to get this right." (12:56)

## Tools & references mentioned
- GenSX
- Palumi
- OpenAI
- ChatGPT
- Gemini
- SQS
- Airflow
- Temporal
- React
- ECS
- Redis
- Next.js
- Vercel
- Deep Research
- GitHub

## Who should watch
- You are building an LLM product whose jobs take longer than a normal HTTP request and need to continue after a user leaves the page.
- Your team is adding queues, polling, retries, and streaming around a serverless setup and wants a concrete architecture for those pieces.
- You need agent workflows with intermediate status, tracing, resumability, or independently scalable API and compute layers.

## Related talks

- [12-Factor Agents: Patterns of Reliable LLM Applications](https://aietalks.com/talks/12-factor-agents-patterns-of-reliable-llm-applications) (Dex Horthy, HumanLayer, 17:06)
- [Your agent architecture has a half-life of 6 months](https://aietalks.com/talks/your-agent-architecture-has-a-half-life-of-6-months) (Dan Farrelly, Inngest, 19:20)
- [A2A & MCP Workshop: Automating Business Processes with LLMs](https://aietalks.com/talks/a2a-mcp-workshop-automating-business-processes-with-llms) (Damien Murphy, Bench, 1:23:14)
- [Building Durable Agents with Workflow DevKit & AI SDK](https://aietalks.com/talks/building-durable-agents-with-workflow-devkit-ai-sdk) (Peter Wielander, Vercel, 1:09:49)
- [From Stateless Nightmares to Durable Agents](https://aietalks.com/talks/from-stateless-nightmares-to-durable-agents) (Samuel Colvin, Pydantic, 22:13)