Enterprise teams should begin with the business problem they need to solve, then decide where AI can change the product or workflow.
2
Evaluations should be designed early with representative examples, because they guide architecture choices and reveal failures that small demos miss.
3
Fine-tuning has costs and can limit performance outside the tuned domain, so teams should first try prompting, caching, retrieval, and architectural changes.
Summary
Alexander Bricken and Joe Bayley describe Anthropic's work with enterprise customers and the technical choices that affect production AI systems. They present Anthropic's API, Claude for Work, and access through AWS Bedrock and Google Cloud Vertex AI. Their examples include products that personalize learning, tax and legal workflows, and Intercom's Finn customer support agent. The main practical advice concerns evaluation. Teams should define representative test cases and success metrics before building a large workflow, then use telemetry, backtesting, and iterative testing to guide changes. The right trade-off depends on the use case. A support agent may need a response within seconds, while a financial research system may justify a longer wait. They also caution against treating fine-tuning as the default solution. Prompting, prompt caching, contextual retrieval, citations, and agentic architectures can change performance, cost, and latency without retraining the model.
Start with the business problem before choosing an AI feature
Joe Bayley says teams should return to the core problem their product is meant to solve instead of stopping at chatbots or summarization. For an onboarding and upskilling platform, a basic approach might summarize course material or answer questions. A stronger product could personalize course content to each employee, make lessons harder when someone moves quickly, or change material based on how the person learns. The point is to ask whether the AI feature improves the underlying outcome, such as getting employees up to speed or helping them progress in their careers.
Anthropic offers several deployment paths for enterprise teams
Bayley describes three ways organizations can use Anthropic models. The API is for businesses that want to embed AI in their own products and services. Claude for Work gives an organization a way to use AI in day-to-day work. Anthropic also works with AWS and Google Cloud, where customers can access the models through Bedrock or Vertex AI and deploy applications in an existing environment. He says support is available whether a customer uses Anthropic directly or accesses the models through a third party.
Applied AI work connects customer implementation with model and product research
Bricken explains that Anthropic's Applied AI team helps with the technical parts of customer use cases. This includes architecture design, evaluations, prompt changes, and other ways to get better results from the models. When customers face difficult domain-specific problems, the team may run a sprint around LLM operations, architecture, or evaluation metrics. The team then helps put the resulting changes into an A/B test and, if they work, into production. Lessons from these projects also go back into Anthropic's product and model research.
Intercom improved Finn through a focused prompt-optimization project
Intercom worked with Anthropic on its Finn customer support agent. The teams started with Finn's hardest prompt and compared it with a prompt developed with Claude. After an initial two-week sprint, they spent about two months optimizing Finn's prompts against Intercom's benchmarks. Bayley says Anthropic's model outperformed the previous model, and Intercom moved forward with it as Finn 2. He cites a result of solving up to 86% of customer support volume, with 51% working out of the box. The model also allowed control over tone and answer length and handled policy-aware answers such as refund policies.
Evaluations should guide the workflow from the beginning
Bayley warns against building a large architecture first and adding evaluations afterward. Evals should direct the team toward a good outcome from the start, or soon after. Small demos and a few queries can create false confidence if they do not use a representative sample. Teams should use telemetry and backtesting, define test cases that resemble real requests, and include unusual but plausible inputs. His example is a customer support website receiving an unrelated question about killing a zombie in Minecraft. The system should be tested on that kind of input so the team knows whether it responds or reroutes it appropriately.
The use case determines the trade-off between intelligence, speed, and cost
Teams should decide their optimization priorities before changing the system. Bayley describes a trade-off among intelligence, cost, and latency, with most organizations able to optimize for only one or two at once. A support agent may need to answer within 10 seconds because a user could leave the page. A financial research agent may be allowed to work for 10 minutes if its answer informs a capital allocation decision. The stakes and time sensitivity should determine whether the team favors faster responses, higher performance, or another metric. User experience choices, such as showing a thinking indicator, can also affect how latency is perceived.
Fine-tuning should follow a clear success criterion
Bayley says fine-tuning is not a universal fix and can affect the model's reasoning in areas outside the tuned domain. Teams often ask for fine-tuning before they have an evaluation set or a definition of success. His recommendation is to try other approaches first, then fine-tune only when the team cannot reach its target in the specific intelligence domain. Teams should justify the added effort and cost. They can continue building the use case with other methods and substitute a fine-tuned model later if testing shows that it is needed.
System architecture can change results without changing the base model
Near the end, Bricken lists several alternatives to relying only on basic prompt engineering. Prompt caching can reduce cost and increase speed while preserving the model's instructions, and he cites a possible 90% cost reduction and 50% speed increase. Contextual retrieval can improve how information reaches the model. Citations can be available out of the box, while agentic architectures change how the whole system works. These choices affect retrieval, processing time, and the final result, so teams should evaluate them alongside prompt changes and fine-tuning.
"Your evals are actually the thing that directs you towards a perfect outcome."13:29
Who should watch
You are choosing between the Anthropic API, Claude for Work, or a cloud deployment and need to understand the practical differences.
Your team has a promising LLM prototype but has not built representative evaluations or agreed on latency, cost, and quality targets.
You are considering fine-tuning and want to compare it with prompting, caching, retrieval, and architecture changes first.
