Production GenAI systems depend on stable input and output specifications, supported by evaluations that test more than a single successful prompt.
2
Embedding search needs access-pattern knowledge, filters, faceting, suitable indexes, and an understanding of how users interact with the product.
3
Speed, user experience, context management, and inference economics often determine whether an AI feature gets used.
Summary
Randall Hunt shares lessons from more than 200 enterprise GenAI implementations at Caylent. He argues that production systems start with a clear definition of inputs, outputs, and return on investment. Models, tools, prompts, and system architecture will change, so teams need evaluations that expose failures across varied data rather than relying on a single successful demonstration. Hunt covers multimodal video search, annotation, embeddings, vector indexes, document processing, generative UI, and context management. He also explains why customer behavior can overturn technical assumptions. Nurses rejected a voice bot because hospitals were noisy, while users in low-connectivity areas needed selected PDF pages rather than large downloads. The talk is practical about cost and performance. Hunt discusses prompt caching, batch inference, output tokens, index storage, and choosing between services such as PostgreSQL, OpenSearch, and Redis. He is also direct that ordinary software should handle calculations instead of asking an LLM to do them.
Production systems need stable input and output specifications
Hunt says the foundation of an AI system is its inputs and outputs. The evaluation layer proves that the system works across more than a one-off prompt or a subjective vibe check. The architecture, language models, and tools are incidental because they will evolve. The underlying reason for the system, its expected result, and its return on investment need to remain clear. He describes this as the part of the application that defines what the system should do, even as models gain new capabilities or change modality. Without that definition, teams can switch technologies without knowing whether the product still meets its purpose.
Multimodal search improves when video is annotated before inference
For Nature Footage, Caylent indexed stock video with Nova Pro-generated descriptions, timestamps, and features, then stored them in Elasticsearch. Frame samples were pooled into multimodal embeddings using Titan v2 so text queries could find images. In a sports application, the team processed audio and video separately, generated transcriptions and embeddings, and identified behaviors with confidence values. Hunt says a small annotation can improve video understanding more than using raw footage alone. Drawing a blue line over the three-point line lets a model answer whether a player crossed it. Meta's SAM 2 can help create such annotations.
Embeddings alone do not make a useful query system
Hunt says teams need to understand access patterns and how people will use the product, rather than adding evaluations and embeddings without studying the application. Embeddings do not provide all the behavior needed for a query system. Users may need faceted search and filters on top of semantic results. That is why he prefers systems such as OpenSearch and PostgreSQL, which can combine vector search with other query mechanisms. He also stresses that speed affects adoption. A slow system can sometimes remain usable with caching or a user interface that occupies the user while inference runs, but an experience that is both slow and expensive will not be used.
Prompt engineering can replace more elaborate model customization
Hunt says prompt engineering has worked far better for his team than he expected as models improved. He had previously expected more fine-tuning and related methods to be needed. Moving the same prompts from Claude 3.5 to Claude 3.7 caused regressions in some evaluations, while the move from Claude 3.7 to Claude 4 produced no regressions in the cases he discussed. He describes Claude 4 as faster, better, and cheaper across virtually every use case he observed. He also points out that prompt placement affects caching. Dynamic information placed after the instructions can allow more effective caching than putting it near the top.
Evaluations can start with a simple human judgment
Hunt describes an evaluation process that starts with a vibe check, the first test someone runs against an application. Teams can vary the input data and quickly turn those examples into an evaluation set. Later, they can add formal metrics. A metric does not have to be a complex benchmark score. It can be a boolean that asks whether the inference succeeded. This makes it easier to begin testing and keep iterating. The purpose is to expose behavior across changing inputs, rather than treating one impressive result as evidence that the full system is reliable.
User behavior can invalidate an otherwise reasonable interface
Hunt gives examples where deployment depended on observing actual users. A hospital system first received a voice bot for nurses, but the nurses disliked it because hospitals were loud and speech transcription picked up surrounding voices. They preferred a standard chat interface. In another case, users in remote areas needed summaries of long PDFs but could not easily download 200-megabyte documents. The team sent a text summary and an image of the relevant page instead. These examples show why teams need to understand daily working conditions before deciding which interaction model to build.
Context and economics shape whether an AI feature is viable
Hunt says applications can differentiate through context about the user, such as the page they are viewing, their browsing history, and other relevant information. Teams should find the minimum context needed for a correct inference, remove irrelevant material, and improve that context over time. They also need to track output-token costs, use prompt caching, consider tool usage and batch processing, and choose an appropriate model. Hunt says Bedrock batch inference costs 50 percent less than the normal model inference price. He warns against using an LLM for arithmetic because it is an expensive way to calculate.
"What will not evolve and change is your fundamental definition and specification of what are your inputs and what are your outputs."08:54
Who should watch
You are moving a GenAI prototype into an enterprise product and need a practical way to define inputs, outputs, and evaluations.
Your team is choosing between vector-search systems, model options, or more complex orchestration and needs advice grounded in deployment constraints.
You are designing an AI interface for users with noisy workplaces, poor connectivity, or unfamiliar interaction patterns.
