Companies need a coherent data strategy and straight-through processing before agents can deliver useful work.
2
Teams should compare the return and cost of building now with the possibility that a better model will solve the problem more cheaply later.
3
AI can make prototypes and internal applications cheap to replace, but production infrastructure, security, monitoring, and core system ownership still need careful human control.
Summary
John Stecher and Scott Worland discuss how to build AI systems while models keep changing. Stecher puts data strategy first: companies need to move information out of transactional systems into forms that agents can access and manage. He also argues that ordinary workflow automation still matters, with agents adding judgment where people previously made decisions. Worland describes how Norm AI initially built a heavily structured workflow around models. A large collection of expert-labeled evaluation examples let the team test newer agent approaches and discover that much of its code had become unnecessary. Both speakers recommend making deliberate choices about what to build now and what to defer until models improve. Fast proofs of concept are useful, but production brings infrastructure, cybersecurity, support, and monitoring requirements. Norm keeps human review around core primitives and sensitive data while leaving room to replace other parts of the platform. Stecher says internal applications can now be rebuilt when rewriting costs less than maintaining them.
John Stecher says companies often want to build many agents before fixing how their data is stored. He argues that a cohesive data strategy must move information out of transactional systems into places where an agent can access, manage, and recall it. Blackstone combines proprietary and public data, so this movement and organization are a major part of its AI work. Without that foundation, Stecher expects agent projects to fail even when the models themselves are capable. He treats data preparation as a condition for useful agent deployment, rather than as a later cleanup task.
Straight-through processing still removes more work than agents alone
Stecher says companies focus too heavily on agents as the main differentiator. He uses agents to understand existing workflows, write code, and optimize processes for straight-through processing. Agents can then add intelligence where humans previously made decisions in otherwise repetitive work. He also points to software maintenance tasks such as writing functional and performance tests, versioning libraries, and renovating source code. Offloading these tasks has increased the delivery velocity of his teams. His approach starts with simplifying the whole workflow, then placing agent-based judgment where it is needed.
Norm's early workflow scaffolding became an obstacle as agents improved
When Norm AI built its first production systems, the team added structure at many levels. It controlled how content entered model context, sequenced model calls, checked consistency, and distilled results into a final output. The system resembled a large map-reduce workflow and produced accurate, consistent results for regulatory work. Norm also created tens of thousands of hand-labeled examples from experts to regression-test it. When newer agent approaches became practical, the team ran them against that benchmark and saw that much of its carefully written code was obsolete. The evaluation set made the comparison visible.
Evaluation sets are needed when models, data, or harnesses change
Stecher says financial services has a very high accuracy requirement for agent outputs. Model versions can produce different answers, and changes in the harness or the underlying data can also change results. He recommends maintaining evaluation sets that teams can run repeatedly to measure system health, accuracy, consistency, and repeatability. Worland describes a similar approach at Norm, where lawyers help create benchmarks and evaluations. Instead of defining a roadmap only by which tasks an agent will perform, the team uses scorecards to measure whether agents and humans complete those tasks successfully across several dimensions.
Teams should sometimes wait for a better model when the business need is weak
Stecher evaluates proposed work by comparing its expected return with the true cost of building it today. If a better model and more compute are likely to solve the problem in six months, and there is no pressing business value from releasing it now, Blackstone may defer the work. He says the team has made this choice before. The decision is not permanent postponement. Stecher also warns that teams can wait forever for new models, so he encourages engineers to build interactive proofs of concept quickly, put them in front of people, iterate, and throw them away when they are not useful.
Norm divides investment between durable foundations and replaceable model-layer code
Worland says the choice is a gradient rather than a binary decision between building for today's models and preparing for the frontier. Norm invests heavily in parts such as data capture, data distillation, model state, benchmarks, and evaluations. It invests more carefully in agent primitives and other systems that may need to change as models improve. The team wants conduits through the platform so new frontier capabilities can enter without being blocked by rigid architecture. Its scorecards also let the team measure general improvements without returning to the earlier, heavily structured workflow.
A proof of concept is fast, but production work remains difficult
Stecher separates a clickable, usable proof of concept from a production system. Current development tools can take a team from an idea and a plan to a working prototype quickly, but that prototype has not passed the checks needed in an enterprise. Production requires infrastructure that runs inside the company's environment, cybersecurity review, supportability, and software that is traceable and connected to site reliability monitoring. Stecher says teams underestimate these requirements, especially when they are selling AI tools to financial services companies. Better agents may help with this work, but he does not consider it solved.
Human ownership stays around sensitive core systems
Worland says some parts of Norm's codebase still require human oversight for design, construction, ownership, and approval of changes. These core primitives can support other software, but that software cannot change them without human review. Norm gives data security priority over speed because it handles sensitive information. A convincing demo can therefore be alarming if reaching that capability would expose systems the company is not ready to secure. The platform invests heavily in these protected areas while keeping other parts easier to replace as models improve.
Internal applications can be cheaper to replace than to maintain
Stecher contrasts today's software economics with those of a decade ago. Previously, building an application took so much time, money, and engineering effort that teams expected to keep it running for years. With AI-assisted development, many internal applications can be built for a particular purpose and then discarded when that purpose changes. He suggests asking whether rewriting an application costs less than continuing to version and maintain it. That question would once have seemed unreasonable to a finance leader. He now considers it a normal part of deciding where new software can create more value.
"If they don't have a cohesive data strategy that's moving stuff out of transactional systems into something that can be accessed and easily managed and recalled from an agent perspective, it is going to fail."John Stecher01:50
Who should watch
You are deciding whether an AI feature should be built now or deferred until a stronger model is available.
Your team has a promising agent prototype but still needs to handle enterprise infrastructure, cybersecurity, monitoring, and support.
You are rebuilding an AI platform and need a way to protect sensitive core systems while keeping model-dependent code replaceable.