Sierra treats every agent as a product that needs its own developer platform and operations process.
2
The agent development life cycle turns production conversations into reported issues, tests, releases, and progressively larger test suites.
3
AI can speed up every stage of the life cycle, while traditional software remains useful for deterministic, fast, and inexpensive tasks.
Summary
Zack Reneau-Wedeen describes how Sierra builds customer-facing agents for brands such as Chubbies and SiriusXM. He compares large language models with traditional software: models are flexible and capable of reasoning, but they are also slow, expensive, and unpredictable. Sierra therefore applies a software development life cycle adapted for agents. Teams inspect conversations in production, report failures or opportunities, turn them into issues and tests, and ship releases as those tests pass. An agent may start with a small test set and grow to hundreds or thousands of tests. Sierra is adding AI to each stage to increase the speed of this work. The process also has to absorb model upgrades, reasoning models, multimodal inputs, and voice. Reneau-Wedeen argues that the same underlying agent can respond across channels, with channel-specific phrasing and latency choices. He is candid that agent quality comes from repeated production refinement rather than configuration alone.
AI systems feel unreliable because their inputs and outputs are nondeterministic
Reneau-Wedeen recalls testing early Google Lens during walks through New York. Plant identification sometimes worked and sometimes missed the answer, which made the experience feel like a slot machine. He connects that experience to current AI development: a system may work five times in a row and fail on the sixth, because either the input or the output can vary. This unpredictability is part of building with AI, and it changes how teams need to improve and test their systems.
Long-term improvement comes from repeating a development process
Reneau-Wedeen uses Google Lens as an example of progress through consistent, step-by-step iteration over a decade. The product expanded from identifying plants to searching and shopping from images, translating non-Latin writing, and helping with homework. He connects this progress to the software development life cycle: teams need a process to implement, test, maintain, analyze, and design repeatedly, so that improvement does not cause regressions.
Sierra treats an agent as a product with engineering and operations around it
The Chubbies agent, Duncan Smothers, can answer sizing questions, recommend products, check inventory, track packages, and issue refunds. These actions illustrate Sierra's definition of an autonomous agent, which takes actions instead of only answering questions. Reneau-Wedeen says an agent cannot be built by dragging together boxes. Sierra works with customers through dedicated agent engineering and agent product management teams, much as a company would work on its website or mobile app.
Agent development combines language models with traditional software
Reneau-Wedeen compares large language models with conventional software. Traditional software is deterministic, fast, cheap, rigid, and governed by predictable logic. Language models are flexible, creative, and able to reason through problems, but they can also be nondeterministic, slow, and expensive. Sierra's methodology tries to use the strengths of both. The agent development life cycle resembles a software life cycle, but each stage is refined through customer use in production.
Production conversations become issues, tests, and releases
Sierra customers can use Experience Manager to inspect every conversation and view reports about agent performance. If Duncan gives incorrect inventory information, a reviewer can report that it called one warehouse but failed to call another, or that a call timed out. The report becomes an issue, the issue leads to a test, and a passing test can be included in a new release. Reneau-Wedeen says agents may grow from a handful of tests at launch to hundreds and then thousands as they improve.
The process also captures chances to exceed a customer's request
Quality work is not limited to fixing mistakes. Reneau-Wedeen gives an example from Chubbies, where agents have a budget for delighting customers. If a product is unavailable online, Duncan could potentially use DoorDash to obtain the shorts from a retail location. This turns an unusual or especially helpful interaction into part of the development process, rather than treating the agent as complete once it avoids errors.
Scale and new model capabilities increase the value of disciplined change management
Sierra used to perform the life cycle manually while learning what worked at each stage. The company is now adding AI to the stages to speed up improvement. Reneau-Wedeen says the process becomes more valuable as customers grow from hundreds of thousands of requests to tens of millions. Changes also come from outside the agent itself, including model upgrades, reasoning models, and multimodality. He describes reasoning models as a force multiplier across development, testing, quality assurance, and the steps between them.
One agent platform can adapt across text, web, and voice
Sierra launched voice generally available in October, and SiriusXM uses its voice capability to answer customer calls immediately. Reneau-Wedeen compares the design to responsive web development. Instead of maintaining separate agents for each channel, the same platform and agent code can respond to the channel and modality a customer uses. Teams can still customize phrasing or parallelize requests to reduce latency, but the underlying agent remains shared.
"How would you do if someone just passed you transcribed text of your conversation partner with a few hundred milliseconds of delay and then you had to respond on the spot?"17:27
Who should watch
You are building a customer-facing agent and need a practical process for turning failures in production into fixes and regression tests.
Your team is moving an agent from chat to voice or other modalities and needs to think about shared code, latency, and channel-specific behavior.
You are evaluating whether agent work needs more than prompts and workflows, including engineering, product management, quality assurance, and release operations.
