# How to Improve Your Agents: Academic Lit Review

Joe, Columbia University and Arklex AI | AI Engineer Summit 2025 | 39:02

Source: https://www.youtube.com/watch?v=UOsOfLnAX3Y
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/how-to-improve-your-agents-academic-lit-review
Published: 2025-02-22
Tags: agents, memory, planning, reinforcement-learning

## TL;DR
- AI agents combine perception, reasoning, reflection, and actions, with autonomy ranging from chatbots to systems trusted with a person's credentials.
- Small language models can improve through self-refinement when larger models edit and distill their feedback, though the quality of the larger model sets a ceiling.
- Tree search, reflection, memory, and debate can improve agent performance at test time, while exploratory learning can train models to search, evaluate, and backtrack.

## Summary
Joe defines an AI agent as a system that perceives its environment, reasons about tasks, reflects on its choices, and takes actions. He places agent systems on a five-level autonomy scale, from information-retrieval chatbots to agents trusted to act on a person's behalf. The talk then reviews research methods for improving agents. Self-refinement can help small models, but their feedback may contain noise, so a larger model must edit it into a form the smaller model can use. For sequential tasks such as persuasion, Monte Carlo tree search lets a model simulate possible actions and outcomes before choosing a response. Joe describes reflective MCTS for browser and desktop tasks, adding stored reflections and multi-agent debate for state evaluation. He also presents exploratory learning, which trains a model on search trajectories so it learns to explore and backtrack. The final section connects these methods to Arklex's framework and to benchmarks that include multiple users, agents, tasks, scheduling, security, and human handoffs.

## Key ideas
### An agent perceives, reasons, reflects, and acts on its environment
[01:14](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=74s)
Joe breaks an agent into a process that resembles human interaction with the world. Perception takes in text, images, audio, video, touch, and other sensor information. Reasoning processes that information, breaks a task into steps, and selects tools or actions. Reflection asks whether the previous choice was right and whether the system should go back. Actions include talking to a person or moving from point A to point B. The agent therefore interacts with its environment through actions, rather than only producing a text response.

### Agent autonomy ranges from chatbots to systems trusted with personal credentials
[02:43](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=163s)
Joe compares agent autonomy with the five levels used for self-driving cars. Level one is a chatbot that retrieves information. Level two is agent assist, where a language model suggests a customer-service response but a human approves the message. Level three automates workflows such as meeting bookings or writing job descriptions. Level four lets one person delegate multiple related tasks that share components, knowledge, and resources. Level five is the Jarvis-style system that acts fully on a person's behalf, potentially using their keys and security permissions. Joe separates lower-risk back-office work from higher-risk customer-facing work.

### Self-refinement can make a small model worse when its feedback is noisy
[06:17](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=377s)
Self-refinement asks a model to solve a problem, generate feedback about its answer, and use that feedback to produce an update. Joe says the loop can run several times, but small models such as Llama 7B may generate noisy feedback. That noise can propagate into later corrections, producing the problem he calls "the blind is leading the blind." Feedback from a larger model may also be unusable if its internal reasoning is incompatible with the smaller model. His example is teaching a child with simpler wording instead of grammatical terminology. A larger model can edit the smaller model's feedback before the correction step.

### Synthetic correction traces can teach smaller models to improve themselves
[10:57](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=657s)
Joe describes a process in which a small model attempts a problem, creates feedback, and then has a large model edit that feedback so it fits the smaller model's internal logic. The corrected feedback helps produce an updated answer, and the loop can continue until the mathematical problem is solved. Ground-truth answers allow the team to filter successful trial-and-error trajectories and use them to train the smaller model. On BigBench Hard tasks, Joe reports that three rounds of feedback and answer editing reached 48%. He says on-policy self-improvement data performed better than simple supervised fine-tuning, while the editor model's ability remains a ceiling.

### Tree search lets an agent simulate possible dialogue strategies before acting
[18:22](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=1102s)
For a persuasion dialogue, the agent must choose what to say next while considering the other person's likely response. Joe applies Monte Carlo tree search to this sequential decision problem. A language model proposes a promising action, simulates its outcome, evaluates the action's quality, and updates its estimate over repeated trials. Another language model simulates the user's behavior. Because human responses vary, the system uses an open-loop version that samples possible simulated conversations instead of relying on one fixed response path. In evaluations, the planning system produced more donations and was judged more convincing, natural, and coherent. It also learned to delay the main request and vary emotional and logical appeals.

### Reflective MCTS improves visual agents through memory and debate
[27:00](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=1620s)
Joe extends tree search to computer-use tasks such as clearing a shopping cart, finding products, navigating files, and using applications. The RMCTS method explores a large action space while constructing a search tree. It adds contrastive reflection, which records successes and errors in a vector database after a task. Future tasks retrieve similar reflections based on the current task or computer state. It also uses multi-agent debate to evaluate whether an action is good or bad, asking for reasons on both sides instead of relying on one model's judgment. Joe reports that this method outperformed other search methods on VisualWebArena and was the best non-trained method on OSWorld.

### Exploratory learning trains models to search and backtrack
[33:39](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=2019s)
Imitation learning would train on the best action found in a search tree. Joe's exploratory learning approach treats the tree search as a trajectory and teaches the model how to linearize its traversal. The model learns to explore, evaluate actions, and backtrack when an action does not produce the desired state. His example asks an agent to find a recent coffee maker with a touchscreen and comment that an item is great. After one action fails, the learned process releases it, returns to an earlier point, and tries another action. Joe says this approach further improves the language model when combined with test-time compute.

### Practical agent systems need benchmarks for coordination, security, and handoffs
[36:27](https://www.youtube.com/watch?v=UOsOfLnAX3Y&t=2187s)
Joe says current benchmarks usually test one agent completing one task assigned by one human. Real deployments can put several tasks on the same computer or have multiple humans assigning work to multiple agents. That introduces system problems around scheduling, database interactions, side effects, security, and deciding when to hand control to a person. Arklex's open-source framework combines research such as continuous learning and task decomposition with system, HCI, and security concerns. Joe says the team is working on more realistic benchmarks that measure task completion alongside efficiency and security.

## Notable quotes
- "An agent needs to understand the world, the environment through sensing information from text, image, audio, video, touch and so on." (01:14)
- "We actually instead of improving the results are getting even less results over time and then people may say, oh, these feedbacks generated by these smaller models are not great." (09:27)
- "The idea is very simple: you propose a certain move and then you simulate what are the values or changes after you make this move and then you evaluate in terms of what are the outcomes of this particular move." (20:18)
- "The model actually learned this process, which we find is very surprising and great." (35:17)

## Tools & references mentioned
- Arklex AI
- Columbia University
- LangChain
- CrewAI
- Bill Gates
- Andrew Ng
- Sam Altman
- OpenAI
- GPT-4
- Llama 7B
- Python
- BigBench Hard
- Monte Carlo Tree Search
- VisualWebArena
- OSWorld
- Reddit
- VCode
- Excel
- Mechanical Turk

## Who should watch
- You are building agents and need concrete methods for improving reasoning, reflection, planning, and computer use without immediately retraining a large model.
- Your agent uses a smaller model and you want to understand why self-generated feedback can fail, and how a larger model can edit training traces for it.
- You are designing an agent platform for multiple tasks or users and need to think about scheduling, security, human handoffs, and evaluation beyond task completion.

## Related talks

- [Why (Senior) Engineers Struggle to Build AI Agents](https://aietalks.com/talks/why-senior-engineers-struggle-to-build-ai-agents) (Philipp Schmid, Google DeepMind, 10:40)
- [Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents](https://aietalks.com/talks/break-it-til-you-make-it-building-the-self-improving-stack-for-ai-agents) (Aparna Dhinakaran, Arize, 14:25)
- [Build an AI Research Agent](https://aietalks.com/talks/build-an-ai-research-agent) (Apoorva Joshi, MongoDB, 27:33)
- [AI Engineering 201: The Rest of the Owl](https://aietalks.com/talks/ai-engineering-201-the-rest-of-the-owl) (Charles Frye, Full Stack LLM Bootcamp, 56:57)
- [How to Build AI Agents that Actually Work](https://aietalks.com/talks/how-to-build-ai-agents-that-actually-work) (Patrick Dougherty, Rosco, 17:44)