# Rethinking how we Scaffold AI Agents Rahul Sengottuvelu, Ramp | AI Engineer Summit 2025 | 16:32 Source: https://www.youtube.com/watch?v=-rsTkYgnNzM Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/rethinking-how-we-scaffold-ai-agents Published: 2025-03-19 Tags: agents, cost, tool-use ## TL;DR - Systems that can use more computation tend to improve more than rigid systems built from fixed rules. - Ramp's CSV migration agent works better when an LLM writes and checks transformation code across many parallel attempts. - Future software may use an LLM as the backend that interprets user actions, calls tools, accesses data, and renders the interface. ## Summary Rahul Sengottuvelu argues that agent scaffolding should follow the bitter lesson: prefer systems that improve when more computation and better models become available. He compares fixed code, hybrid pipelines, and an approach where an LLM decides when to write and run classical code. Ramp's CSV migration agent illustrates the tradeoff. Instead of maintaining parsers for many card providers or classifying every column with hand-built logic, the system gives an LLM a code interpreter, data access, a target format, and a verifier. Running many attempts costs more inference but reduces engineer time and handles unfamiliar schemas. Sengottuvelu then extends the idea to software architecture. In his experimental email client, the LLM acts as the backend, using tools and database access to decide what UI to render after each user action. The demo is slow and unreliable, and he is open about that. His point is that model improvements could make this architecture increasingly practical. ## Key ideas ### Agent scaffolding should improve when it gets more computation [02:12](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=132s) Sengottuvelu reduces the bitter lesson to a design rule: systems that scale with compute beat systems that do not. A rigid system is fixed and deterministic, while another system can think more or search more when given additional resources. The second kind can keep improving without requiring engineers to redesign its logic. He connects this to chess, Go, computer vision, and Atari games. Carefully engineered systems can win when compute is fixed, especially when they encode human reasoning into features. Once search and computation scale, general methods tend to win. His recommendation is to build agents that can take advantage of more inference and stronger models instead of locking behavior into handcrafted rules. ### More model intelligence should let teams delete scaffolding [00:58](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=58s) His experience building customer-support automation led him to watch scaffolding disappear as models improved. Early models had small context windows, weak reasoning, and poor reliability, so his team wrote extensive code around them. As the models became more capable, some of that code had to be removed. Jsonformer followed a similar pattern. Models were bad at producing JSON, so the library constrained and forced their output. Sengottuvelu uses these examples to argue that scaffolding should be designed with future model improvements in mind. Code that compensates for a temporary model weakness may become a burden later. The practical question is which parts of an agent should remain fixed as intelligence increases. ### A fixed parser for every CSV provider creates an expanding maintenance burden [04:40](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=280s) Ramp's switching report takes transaction CSVs from third-party card providers and converts arbitrary schemas into a format Ramp understands. The simplest design is to support the 50 most common vendors with manually written parsers. That approach works, but engineers must inspect each provider's files, write parsing code, and repair it when a provider changes its format. Sengottuvelu describes this as a deterministic system with a growing amount of maintenance. It handles known cases through explicit code, but it does not generalize naturally to unfamiliar CSV structures. The cost is paid in engineer time, and every new provider or schema variation creates another case for the team to maintain. ### A hybrid CSV pipeline uses models for classification while keeping most logic deterministic [05:56](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=356s) The next design adds language-model or embedding calls to a conventional script. The system examines each incoming CSV column and predicts whether it is a date, transaction, amount, merchant name, or user name. It then maps those classifications into Ramp's target schema. This is more general than writing a parser for every provider, but most of the work still happens in classical code. The model is used as a fuzzy component inside a larger fixed pipeline. Sengottuvelu presents this as a reasonable intermediate design, while showing that it still assumes engineers know where the model belongs and how the surrounding workflow should operate. ### Giving an LLM tools and verification allows it to discover the CSV transformation [06:57](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=417s) Ramp also tried giving the CSV directly to an LLM with a code interpreter. The model could inspect selected rows, use Python packages such as pandas, write transformation code, and produce the requested output format. A unit test and verifier told it whether the result worked. One attempt was not reliable enough, but running 50 attempts in parallel made the system likely to work across many formats. Sengottuvelu says this used roughly 10,000 times more compute than the first approach, yet the cost could still be less than a dollar per transaction. The tradeoff favors computation because engineer time is scarce and a failed migration can cost Ramp more than additional inference. ### The most scalable architecture lets the LLM decide when to use classical code [08:30](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=510s) Sengottuvelu compares three agent architectures. The first is entirely classical code. The second calls an LLM from a conventional program for tasks such as similarity or classification. The third reverses the relationship: the LLM handles most of the work and writes or runs classical code when it needs it. In that design, the model may generate pandas or Python code and move into deterministic computation for a specific operation. He argues that more of Ramp's codebase is moving toward this pattern because the fuzzy part benefits directly from better models. Even if the engineers stop changing their system for a year, the model providers continue improving the component doing more of the work. ### An LLM backend could interpret requests and render a UI from tools and data [10:49](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=649s) The talk's final architectural proposal replaces a conventional web backend with an LLM. A traditional Gmail-like application sends static frontend files to the browser, receives user actions, calls backend endpoints, queries a database, and returns results. In Sengottuvelu's alternative, the LLM has access to a code interpreter, network requests, and a database. It receives the Gmail token and a description of the available capabilities, then decides what interface to render. When the user clicks an email, the frontend sends the clicked text and ID back to the same model. The model can fetch the message and choose the next UI, much as a web framework would. ### The proposed software model is slow and barely works today [15:55](https://www.youtube.com/watch?v=-rsTkYgnNzM&t=955s) Sengottuvelu does not present the LLM-driven email client as a finished product. The live demo is slow because each page and interaction involves a language-model session with access to Gmail and tools. The model renders markdown, chooses available email actions, and attempts to respond to clicks, but the interaction becomes unreliable during the demonstration. He says this kind of software barely works today. His claim is narrower than a prediction that it will definitely replace conventional applications. He wants engineers to consider the possibility that improving models could make this architecture practical, especially if they design systems that can benefit from those improvements rather than hard-coding every behavior. ## Notable quotes - "systems that scale with compute beat systems that don't" (02:12) - "what is truly scarce in the world is engineer time" (07:53) - "the back end is the LM" (11:31) - "this kind of software barely works today" (16:00) ## Tools & references mentioned - Ramp - Cohere.io - Jsonformer - OpenAI - Gmail - pandas ## Who should watch - You are building an agent with a large amount of hand-written routing, parsing, or policy code and want to know which parts may become liabilities as models improve. - Your system handles unfamiliar inputs, such as third-party CSVs, and you can trade additional inference cost for less engineering and maintenance work. - You are exploring model-driven application backends and want to see an honest example that is slow and unreliable rather than a polished product demo. ## Related talks - [Scaling Agents for Gen AI Products](https://aietalks.com/talks/scaling-agents-for-gen-ai-products) (Anju Kambadur, Bloomberg, 19:38) - [Building Applications with AI Agents](https://aietalks.com/talks/building-applications-with-ai-agents) (Michael Albada, Microsoft, 15:50) - [Don't Build Slop (4 Levels of AI Agent Maturity)](https://aietalks.com/talks/dont-build-slop-4-levels-of-ai-agent-maturity) (Ara Khan, Cline, 18:52) - [Scaling the Next Paradigm of Heterogeneous Intelligence](https://aietalks.com/talks/scaling-the-next-paradigm-of-heterogeneous-intelligence) (Adrian Bertagnoli, Callosum, 15:13) - [Code Generation and Maintenance at Scale](https://aietalks.com/talks/code-generation-and-maintenance-at-scale) (Morgante Pell, Grit, 18:54)