# POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments

Randall Hunt, Caylent | AI Engineer World's Fair 2025 | 19:16

Source: https://www.youtube.com/watch?v=vW8wLsb3Nnc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/poc-to-prod-hard-lessons-from-200-enterprise-genai-deployments
Published: 2025-07-23
Tags: cost, design, evals, rag

## TL;DR
- Production GenAI systems depend on stable input and output specifications, supported by evaluations that test more than a single successful prompt.
- Embedding search needs access-pattern knowledge, filters, faceting, suitable indexes, and an understanding of how users interact with the product.
- Speed, user experience, context management, and inference economics often determine whether an AI feature gets used.

## Summary
Randall Hunt shares lessons from more than 200 enterprise GenAI implementations at Caylent. He argues that production systems start with a clear definition of inputs, outputs, and return on investment. Models, tools, prompts, and system architecture will change, so teams need evaluations that expose failures across varied data rather than relying on a single successful demonstration. Hunt covers multimodal video search, annotation, embeddings, vector indexes, document processing, generative UI, and context management. He also explains why customer behavior can overturn technical assumptions. Nurses rejected a voice bot because hospitals were noisy, while users in low-connectivity areas needed selected PDF pages rather than large downloads. The talk is practical about cost and performance. Hunt discusses prompt caching, batch inference, output tokens, index storage, and choosing between services such as PostgreSQL, OpenSearch, and Redis. He is also direct that ordinary software should handle calculations instead of asking an LLM to do them.

## Key ideas
### Production systems need stable input and output specifications
[07:58](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=478s)
Hunt says the foundation of an AI system is its inputs and outputs. The evaluation layer proves that the system works across more than a one-off prompt or a subjective vibe check. The architecture, language models, and tools are incidental because they will evolve. The underlying reason for the system, its expected result, and its return on investment need to remain clear. He describes this as the part of the application that defines what the system should do, even as models gain new capabilities or change modality. Without that definition, teams can switch technologies without knowing whether the product still meets its purpose.

### Multimodal search improves when video is annotated before inference
[02:03](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=123s)
For Nature Footage, Caylent indexed stock video with Nova Pro-generated descriptions, timestamps, and features, then stored them in Elasticsearch. Frame samples were pooled into multimodal embeddings using Titan v2 so text queries could find images. In a sports application, the team processed audio and video separately, generated transcriptions and embeddings, and identified behaviors with confidence values. Hunt says a small annotation can improve video understanding more than using raw footage alone. Drawing a blue line over the three-point line lets a model answer whether a player crossed it. Meta's SAM 2 can help create such annotations.

### Embeddings alone do not make a useful query system
[12:14](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=734s)
Hunt says teams need to understand access patterns and how people will use the product, rather than adding evaluations and embeddings without studying the application. Embeddings do not provide all the behavior needed for a query system. Users may need faceted search and filters on top of semantic results. That is why he prefers systems such as OpenSearch and PostgreSQL, which can combine vector search with other query mechanisms. He also stresses that speed affects adoption. A slow system can sometimes remain usable with caching or a user interface that occupies the user while inference runs, but an experience that is both slow and expensive will not be used.

### Prompt engineering can replace more elaborate model customization
[13:54](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=834s)
Hunt says prompt engineering has worked far better for his team than he expected as models improved. He had previously expected more fine-tuning and related methods to be needed. Moving the same prompts from Claude 3.5 to Claude 3.7 caused regressions in some evaluations, while the move from Claude 3.7 to Claude 4 produced no regressions in the cases he discussed. He describes Claude 4 as faster, better, and cheaper across virtually every use case he observed. He also points out that prompt placement affects caching. Dynamic information placed after the instructions can allow more effective caching than putting it near the top.

### Evaluations can start with a simple human judgment
[14:49](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=889s)
Hunt describes an evaluation process that starts with a vibe check, the first test someone runs against an application. Teams can vary the input data and quickly turn those examples into an evaluation set. Later, they can add formal metrics. A metric does not have to be a complex benchmark score. It can be a boolean that asks whether the inference succeeded. This makes it easier to begin testing and keep iterating. The purpose is to expose behavior across changing inputs, rather than treating one impressive result as evidence that the full system is reliable.

### User behavior can invalidate an otherwise reasonable interface
[16:40](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=1000s)
Hunt gives examples where deployment depended on observing actual users. A hospital system first received a voice bot for nurses, but the nurses disliked it because hospitals were loud and speech transcription picked up surrounding voices. They preferred a standard chat interface. In another case, users in remote areas needed summaries of long PDFs but could not easily download 200-megabyte documents. The team sent a text summary and an image of the relevant page instead. These examples show why teams need to understand daily working conditions before deciding which interaction model to build.

### Context and economics shape whether an AI feature is viable
[17:40](https://www.youtube.com/watch?v=vW8wLsb3Nnc&t=1060s)
Hunt says applications can differentiate through context about the user, such as the page they are viewing, their browsing history, and other relevant information. Teams should find the minimum context needed for a correct inference, remove irrelevant material, and improve that context over time. They also need to track output-token costs, use prompt caching, consider tool usage and batch processing, and choose an appropriate model. Hunt says Bedrock batch inference costs 50 percent less than the normal model inference price. He warns against using an LLM for arithmetic because it is an expensive way to calculate.

## Notable quotes
- "Generative AI is not the magical pill that solves everything that a lot of people seem to think it is." (00:46)
- "What will not evolve and change is your fundamental definition and specification of what are your inputs and what are your outputs." (08:54)
- "Embeddings alone do not a great query system make." (12:31)
- "Nurses hate voice bots because hospitals are loud and noisy and the voice transcription is not very good." (17:17)
- "Don't do math in an LLM. It is the most expensive possible way of doing math." (17:35)

## Tools & references mentioned
- Caylent
- BrainBox AI
- Simmons
- Nature Footage
- Nova Pro
- Amazon Titan v2 multimodal embeddings
- Elasticsearch
- PostgreSQL
- pgvector
- OpenSearch
- Amazon Bedrock
- Amazon SageMaker
- AWS Trainium
- AWS Inferentia
- Neuron SDK
- Claude
- Claude 3.5
- Claude 3.7
- Claude 4
- Llama
- DeepSeek
- Mistral
- Redis
- MemoryDB
- HNSW
- SAM 2
- Meta
- CloudZero
- Amazon SNS
- Amazon End User Messaging
- ShurePath
- DSPy
- Anthropic
- Transformer paper
- NASA
- MongoDB
- SpaceX

## Who should watch
- You are moving a GenAI prototype into an enterprise product and need a practical way to define inputs, outputs, and evaluations.
- Your team is choosing between vector-search systems, model options, or more complex orchestration and needs advice grounded in deployment constraints.
- You are designing an AI interface for users with noisy workplaces, poor connectivity, or unfamiliar interaction patterns.

## Related talks

- [The Build-Operate Divide: Bridging Product Vision and AI Operational Reality](https://aietalks.com/talks/the-build-operate-divide-bridging-product-vision-and-ai-operational-reality) (Chris Hernandez, Chime & Jeremy Silva, Freeplay, 12:50)
- [Productionizing GenAI Models](https://aietalks.com/talks/productionizing-genai-models) (Lukas Biewald, Weights & Biases, 22:36)
- [The Production AI Playbook: Deploying Agents at Enterprise Scale](https://aietalks.com/talks/the-production-ai-playbook-deploying-agents-at-enterprise-scale) (Sandipan Bhaumik, Databricks, 37:06)
- [AI Leadership](https://aietalks.com/talks/ai-leadership-l89gzwei) (Grace Isford, Lux Capital & Hamel Husain, Parlance Labs & Greg Ceccarelli, SpecStory & Don Bosco Durai, Privacera & Colin Flaherty, Augment Code & Stephen Chin, Neo4j & Jonathan Lowe, Pfizer & Bruno Passos, Booking.com & Beyang Liu, Sourcegraph & Waseem Alshikh, Writer & Prashant Mital & Toki Sherbakov, OpenAI & Shirsha Chaudhuri, Thomson Reuters & Aparna Dhinkaran, Arize & Diamond Bishop, Datadog & Paul Gilbert, Arista Networks & Alexander Bricken & Joe Bailey, Anthropic & Heath Black, SignalFire & Xiaofeng Wang, LinkedIn & Douwe Kiela, Contextual AI & Barr Yaron, Amplify Partners, 8:03:36)
- [Anthropic in the Enterprise](https://aietalks.com/talks/anthropic-in-the-enterprise) (Alexander Bricken & Joe Bayley, Anthropic, 20:55)