# The Unreasonable Effectiveness of Prompt Learning

Aparna Dhinakaran, Arize | AI Engineer | 10:56

Source: https://www.youtube.com/watch?v=pP_dSNz_EdQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-unreasonable-effectiveness-of-prompt-learning
Published: 2025-12-23
Tags: agents, coding-agents, evals, prompt-engineering, reinforcement-learning

## TL;DR
- Coding agents depend heavily on long system prompts that teams repeatedly revise as they discover failure cases.
- Prompt learning uses evaluation feedback in English to update an agent's rules without changing the underlying model weights.
- On a 150-example SWE-bench Lite test, the approach improved Claude Code by 5% and Cline by 15% in resolved GitHub issues.

## Summary
Aparna Dhinakaran argues that teams building coding agents should spend more attention on learning system prompts. Agents such as Claude Code and Cline already rely on editable rules files, but those rules are often maintained by hand. Her approach applies reinforcement-learning ideas to prompts rather than model weights. The agent attempts a software-engineering task, runs unit tests, and passes the problem, patch, tests, and result to an LLM judge. The judge explains why the attempt failed. A meta-prompt then turns those explanations into new rules, which are appended to the agent's system prompt before another benchmark run. On 150 SWE-bench Lite examples, the process improved resolved issues by 5% for Claude Code and 15% for Cline. Dhinakaran also compares the method with DSPy prompt optimization. She says both use English feedback, while her approach needed fewer loops because the evaluation prompts produced more useful explanations.

## Key ideas
### Coding agents spend substantial effort learning and maintaining system prompts
[00:13](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=13s)
Dhinakaran opens by pointing out that frontier coding models receive attention, while the system prompts behind coding agents receive less attention. She shows a comparison involving Claude, Cursor, and Cline, including the length of their prompts. These prompts are not static documents. Teams repeatedly revise them as they learn what helps an agent succeed. Andrej Karpathy described this process as system prompt learning, where humans give feedback in English and the agent's instructions change for the next attempt. Dhinakaran compares it to the notes written by the character in the film Memento, which preserve lessons for future actions.

### Prompt learning gives the agent explanations instead of only a reward score
[02:17](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=137s)
Dhinakaran contrasts ordinary reinforcement learning with prompt learning through a student taking exams. In the reinforcement-learning version, the student receives a scalar result such as 70%, 80%, or 90% and has to infer how to improve. She says this can be sample inefficient, time intensive, and data hungry, with a substantial data-science setup. In the prompt-learning version, the student receives an explanation of which answers were wrong, which concepts were missed, and what to study. The agent uses that English feedback to change its instructions. This keeps the learning process focused on prompts while leaving the model itself unchanged.

### Editable rules files give coding agents a place to store learned behavior
[03:49](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=229s)
The experiment starts with Claude Code and Cline, each of which has a base system prompt and a place where repository-specific rules can be added. Dhinakaran refers to Claude's rules file and Cline's rules, describing both as initially empty in the setup. The team gives the agent a software-engineering problem and lets it generate a patch. The learned instructions are then appended to the existing system prompt. The experiment therefore changes what the agent is told to do, rather than fine-tuning the model or replacing the coding agent.

### The learning loop turns test failures into new system-prompt rules
[05:07](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=307s)
The loop begins with a coding task from SWE-bench Lite. The agent produces a solution, and unit tests determine whether it works. An LLM judge receives the problem statement, the agent's solution, the tests, and the test result. It reports whether the attempt passed and explains why it failed. The team asks about recurring issues, such as parsing errors or problems with particular libraries. A meta-prompt combines the original system prompt, the current rules, the task input, the judge's evaluation, and the explanation. It then generates revised rules describing what the agent should avoid or do differently.

### Evaluation quality determines how useful prompt learning becomes
[06:44](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=404s)
Dhinakaran calls the LLM-as-a-judge evaluation the most important part of the process because it supplies the explanation used for the next prompt revision. The judge is asked to produce more than a pass or fail result. It must explain the concrete cause of the failure and identify patterns across attempts. The team reviews these categories and feeds them into the meta-prompt. Her point is that prompt optimization depends on the quality of the evaluation prompt. A weak evaluation gives the learning loop little useful information, while a specific explanation can become an actionable rule.

### A small set of training examples improved benchmark results without model changes
[08:28](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=508s)
Dhinakaran reports results from 150 SWE-bench Lite examples used as the training data for the prompt-learning process. The updated instructions produced 5% more resolved GitHub issues for Claude Code and 15% more for Cline. She stresses that the models were not fine-tuned and that nothing about the underlying models was changed. The intervention focused on the system prompt. She presents this as a way for teams to use runtime failures and evaluations to improve an agent without building a full model-training pipeline.

### The approach differs from DSPy mainly in the evaluation loop
[09:10](https://www.youtube.com/watch?v=pP_dSNz_EdQ&t=550s)
Dhinakaran compares the method with DSPy's prompt optimizer, which also uses English feedback to modify prompts. In a side-by-side benchmark, she says DSPy required many more loops and rollouts than their approach. She describes the underlying idea as similar, then points to the evaluation prompts as the main practical difference. Her team spent time developing and revising evals so the judge returned useful explanations. Those explanations allowed the system to update the rules with fewer iterations.

## Notable quotes
- "What's not so obvious is how much time is actually spent on the system prompts for those building these coding agents." (00:13)
- "We didn't do anything on fine-tuning. We didn't change the models, anything like that. It was just focused on the system prompt." (05:07)
- "Writing really good evals is, I think, how you get the best kind of insight into what you could do to improve your agents." (06:44)
- "The key thing that was really different here was we spent a lot of time actually developing and iterating on the evals." (09:49)

## Tools & references mentioned
- Arize
- Claude
- Claude Code
- Cursor
- Cline
- SWE-bench Lite
- DSPy
- Andrej Karpathy
- Memento

## Who should watch
- You are building a coding agent and maintain an agents.md file, repository rules, or another hand-edited system prompt.
- Your agent produces useful code inconsistently, and you have unit tests, PR feedback, or runtime evaluations that could explain its failures.
- You are comparing prompt optimization with model fine-tuning and want a loop that changes instructions instead of model weights.

## Related talks

- [Build a Prompt Learning Loop](https://aietalks.com/talks/build-a-prompt-learning-loop) (SallyAnn DeLucia & Fuad Ali, Arize, 52:08)
- [Prompt Engineering is Dead](https://aietalks.com/talks/prompt-engineering-is-dead) (Nir Gazit, Traceloop, 14:19)
- [Building with Anthropic Claude: Prompt Workshop](https://aietalks.com/talks/building-with-anthropic-claude-prompt-workshop) (Zack Witten & Jamie Neuwirth, Anthropic, 1:34:56)
- [The Prompt is the Platform](https://aietalks.com/talks/the-prompt-is-the-platform) (Dominik Tornow, Resonate HQ, Inc, 17:33)
- [Prompt Engineering and AI Red Teaming](https://aietalks.com/talks/prompt-engineering-and-ai-red-teaming) (Sander Schulhoff, HackAPrompt and Learn Prompting, 2:01:05)