# Your Old Prompts Are Making Claude Worse

Lance Martin, Anthropic | AI Engineer NYC 2026 | 1:04:03

Source: https://www.youtube.com/watch?v=ys6YQ2yfBpM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-old-prompts-are-making-claude-worse
Published: 2026-11-06
Tags: caching, cost, evals, reliability

## TL;DR
- Frontier models usually do not need prompts that tell them to think step by step, verify their work, or be maximally thorough, and these instructions can raise cost or reduce performance.
- Prompt caching can cut costs without changing performance when repeated context is kept in a stable prefix, with internal benchmarks showing savings of up to 72%.
- Claude can hill climb on well-designed evaluations, but teams need train and test splits, clear grading, and production-relevant cases to avoid overfitting.

## Summary
Lance Martin argues that many Claude applications are held back by old prompt habits and weak evaluation setups. He recommends auditing prompts for instructions such as "think step by step," verification rituals, thought boosters, and contradictory rules. These can add tokens or confuse newer models. He also explains how multi-turn applications should organize stable context so prompt caching remains effective. His internal benchmarks found cost reductions of up to 72% without a meaningful performance change. Martin then describes evaluation-driven hill climbing. A useful eval should have headroom, improve as model capability or thinking increases, and produce consistent results. Claude can modify prompts and skills against such an eval, but train and test splits are needed to catch overfitting. He gives examples of improving a documentation skill from a 66% starting score and cutting cost by 90% while improving performance. The talk closes with long-running agents, where independent session logs, recoverable sandboxes, and secret vaults make the system safer and easier to recover.

## Key ideas
### Old prompting rituals can waste tokens and confuse newer models
[01:32](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=92s)
Martin says prompts accumulate debt like code. Instructions such as "think step by step," "double-check your work," "be maximally thorough," and explicit scratch pads were useful with earlier models, but frontier models can decide when to think and verify their work without being told. These instructions can make the model spend more tokens and can reduce performance. Contradictory rules are another problem. Earlier models might ignore one instruction and continue, while newer models often become confused. Martin recommends auditing existing prompts before adding more instructions, especially when migrating an application to a newer model.

### Prompt audits can improve both cost and performance
[03:49](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=229s)
Martin describes a synthetic customer-support evaluation with about 50 tickets. He compared a migration from Opus 4.8 to Opus 5.5 while injecting prompt anti-patterns individually. The migration produced some cost and performance gains, but running a prompt audit produced another improvement in both measures. The audit removes verification rituals and boosters that cause Opus 5.5 to use excessive tokens. It also addresses contradictory prompts that can confuse frontier models. Martin recommends the Claude Code commands "checkup prompt audit" for Claude Code configurations and "Claude API prompt audit" for code using Anthropic APIs.

### Prompt caching depends on keeping the prefix unchanged
[05:30](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=330s)
In multi-turn agents, much of the context repeats from one turn to the next. Martin says the prompt cache is much cheaper than base tokens and is also faster, so applications should use it heavily. The cache uses a strict prefix match: even a one-byte change in the prefix invalidates it. He suggests placing system instructions, tools, and globally shared context in the cached portion, followed by project memory, session state, and per-turn messages. Claude Console provides cache-hit diagnostics, and Martin recommends tracking the hit rate because it affects application economics.

### Good evals give Claude room to improve
[08:45](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=525s)
Martin says an evaluation should behave sensibly as model capability and thinking increase. A stronger model should score better, more thinking should generally improve the score, and the evaluation should still have headroom at the frontier. Results should also have low run-to-run variance. Ambiguous tasks can make an evaluation appear to have room for improvement when no model can solve the remaining cases. Martin recommends using the Claude API build-eval skill to inspect raw data, create an eval set, and sanity-check the grader and criteria before starting a hill-climbing loop.

### Train-test checks help expose hill-climbing overfit
[11:47](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=707s)
Claude can autonomously inspect failures and modify an evaluation target, but Martin says overfitting is the main risk. A solution might add OCR because one benchmark task needs it, even though production tasks do not. It might also learn distinctive wording from the eval examples. His hill-climb process splits the evaluation into train and test sets. Claude uses only the train failures to decide what to change, then checks both sets after each round. If train performance rises while test performance stays flat, that is an early signal that the changes are fitting the benchmark rather than improving the application.

### Text surfaces are a practical starting point for hill climbing
[14:57](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=897s)
Martin improved a Claude API skill that was grounded in Anthropic documentation. The initial skill scored 66% on a documentation-based evaluation. He gave Claude the skill text and documentation, then let it inspect failures and revise the skill. He says prompts and skills work well because they are easy to modify and closely connected to model behavior. Agent harnesses are possible targets, but their larger code changes make it harder to tell which modification caused an outcome. Prompts are also easy to overfit, so a simple editing surface does not remove the need for production-relevant tests.

### Hill climbing can search for cheaper model configurations
[16:30](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=990s)
Martin describes a cost-focused hill climb that searches model choice, effort, and prompts while keeping performance at least at parity. In his customer-support example, the process moved from Opus 4.8 at high effort to Opus 5 at low effort, then tuned the prompt after performance fell. The final configuration cut cost by 90% and improved performance. He says a stronger model at lower effort often uses fewer tokens than a weaker model at high effort, which can make it cheaper despite a higher base-token price. A saturated evaluation can still be useful when the only goal is reducing cost while holding performance steady.

### Long-running agents need independent, recoverable components
[18:53](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=1133s)
Martin says agents that work for hours or days require different reliability and security choices. Claude Managed Agents separates the agent harness, which acts as the brain, from sandboxes, which perform actions. The session log is an independent, append-only object, so the system can recover if the harness or sandbox fails. A new sandbox or harness can be started through the orchestrator. Secrets stay in vaults, allowing Claude to reference credentials without seeing the secret values. Martin presents this separation as a way to preserve state and limit exposure during long-running work.

### Open-ended grading needs simple rubrics and repeated checks
[30:04](https://www.youtube.com/watch?v=ys6YQ2yfBpM&t=1804s)
For tasks without deterministic checks, Martin recommends a rubric that a human can understand and apply. He prefers crisp criteria, ideally binary good-or-bad judgments, over arbitrary scales such as zero to five or one to ten. Teams should test the rubric across different models and run replicates to check variance. Pairwise comparison can be easier than assigning an absolute score, since a grader can often choose which of two artifacts is better. He also warns against building an eval only from failures of one model, because model capabilities are uneven and those cases may not remain difficult after a model upgrade.

## Notable quotes
- "Prompts accumulate debt just like code." (01:32)
- "Any even a byte change in your prefix will invalidate the cache." (06:18)
- "The hill climber then autonomously tries flying at 5 low." (17:38)
- "Text is nice, in particular if things like prompts and skills are very nice because it's a very clear place that we know significantly alters and affects model behavior." (26:50)
- "It is not guaranteed to converge at all." (36:36)

## Tools & references mentioned
- Anthropic
- Claude
- Claude Code
- Claude API
- Claude Console
- Claude Managed Agents
- Opus
- Sonnet
- Haiku
- Fable
- LangChain
- DSPy
- Omar
- Hamel
- Andrej Karpathy
- AutoResearch
- RLM
- Braintrust
- LangSmith
- Riza

## Who should watch
- You maintain prompts written for older Claude generations and want to check whether they now add cost or interfere with instruction following.
- Your team is building an eval-driven optimization loop and needs practical guidance on headroom, grading, test splits, and overfitting.
- You are running agents for hours or days and need recoverable state, sandbox isolation, and safer handling of secrets.

## Editor's note

Lance Martin warns that hill climbing can overfit when an evaluation is built from narrow or model-specific failures. Kitaru records real agent runs and lets a person turn what they notice into cohorts of runs, so evaluations can be checked against failures from actual sessions rather than only the cases a single model exposed.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Building with Anthropic Claude: Prompt Workshop](https://aietalks.com/talks/building-with-anthropic-claude-prompt-workshop) (Zack Witten & Jamie Neuwirth, Anthropic, 1:34:56)
- [Principles for Prompt Engineering](https://aietalks.com/talks/principles-for-prompt-engineering) (Karina Nguyen, Anthropic, 55:31)
- [The Unreasonable Effectiveness of Prompt Learning](https://aietalks.com/talks/the-unreasonable-effectiveness-of-prompt-learning) (Aparna Dhinakaran, Arize AI, 10:56)
- [Your Agent Is Wasting Tokens and You Don't Know It](https://aietalks.com/talks/your-agent-is-wasting-tokens-and-you-dont-know-it) (Erik Hanchett, Amazon Web Services, 05:55)
- [Build a Prompt Learning Loop](https://aietalks.com/talks/build-a-prompt-learning-loop) (Sally-Ann DeLucia & Fuad Ali, Arize AI, 52:08)
