An AI agent needs a goal, access to at least one tool, and the ability to reason about when and how to use that tool.
2
Small changes to tool-call inputs and outputs can improve an agent more than fine-tuning the underlying model.
3
Production systems need careful control of authentication, user experience, tool interfaces, and multi-agent coordination.
Summary
Patrick Dougherty describes lessons from rebuilding Rosco around AI agents before the company was acquired by Klarity in 2024. He defines an agent as a system that receives an objective, can call a tool, and can decide autonomously how to use that tool. His central advice is to give agents ways to retrieve information during their work instead of filling prompts with large amounts of background knowledge. He compares GPT-4o, which tends to produce SQL even when the available schema cannot answer the question, with o1, which can recognize that limitation. Dougherty also discusses agent computer interfaces, model-specific response formats, model selection, and why fine-tuning reduced reasoning in his experience. He explains why his team avoided becoming dependent on agent frameworks, especially when user credentials had to flow into data warehouse queries. For multi-agent systems, he recommends a manager agent with focused worker agents and limits teams to roughly five to eight agents.
An agent needs an objective, a tool, and autonomous decisions about tool use
Dougherty defines an AI agent with three conditions. It must receive one specific objective or overarching goal, supplied by a human or another AI. It must have access to at least one tool and receive a response from that tool. It must also reason autonomously about how and when to use the tool. A predefined chain, where one tool always runs before the next, does not meet his definition. The distinction matters because his team designed systems that chose their own retrieval steps while answering questions about enterprise data.
Retrieval tools give the model room to reason about unfamiliar data
Rosco built an agent that searched and queried enterprise data warehouses. Giving the agent every table and column in the prompt often overwhelmed it, especially when it had to write SQL. The agent might choose the wrong table or produce a query that did not execute. Dougherty's team instead exposed smaller tools such as searching tables, getting table details, and profiling a column. The agent could call these tools iteratively, find the relevant fields, and then form its query. He prefers this approach to inserting large retrieved passages into the system prompt.
A reasoning model can admit that a question cannot be answered
Dougherty compares GPT-4o with o1 using a Salesforce-style schema containing accounts, contacts, and opportunities. When asked for customers who churned in the last month, GPT-4o wrote SQL despite having no field that could establish churn. It made assumptions about account types and updates, which could lead an analyst to a wrong result. With the same schema and question, o1 reasoned through the available information and concluded that churn could not be calculated from the schema. Dougherty uses this contrast to argue that agents need permission to investigate and reject impossible tasks.
The agent computer interface can matter as much as the model
Dougherty calls the syntax and structure of tool calls the agent computer interface, or ACI. This includes the arguments sent to a tool and the content and format returned by its API or Python code. His team found that small ACI changes had large effects on consistency. GPT-4o often missed columns in long Markdown search results, including warehouse tables with hundreds or thousands of columns. Switching the response from Markdown to JSON immediately fixed the problem for GPT-4o. Later, Claude worked better when the response used XML. The right format depends on the model and the tool.
Model failures reveal the interface the model expects
Dougherty treats a model's recurring mistakes as evidence about the tool format it expects. If a model repeatedly ignores a JSON schema and supplies an argument in another format, that behavior may indicate how its training data has taught it to structure the call. Adjusting the schema toward that expected format can improve performance. He recommends using a generally intelligent model for the decision about which tool to call next, even if cheaper models handle some individual tools or sub-prompts. He names Claude 3.5 as a strong choice for balancing speed, cost, and decision quality.
Fine-tuning can make an agent follow routines instead of reasoning
Dougherty says fine-tuning was a waste of time for his team. His reasoning is that if the system depends on the agent thinking through its next action, teaching a model a fixed task sequence can work against that goal. In his experience, fine-tuning often reduced reasoning because the model became overfit to performing a particular sequence. He recommends spending that development time on iterating the ACI instead of building a fine-tuned model for the agent.
Production constraints can make an agent framework a serious dependency
Rosco started before libraries such as LangGraph and CrewAI were publicly available, but Dougherty says production requirements kept the team from moving onto them. One difficult requirement was passing an end user's Snowflake permissions into the agent through an OAuth integration. The agent had to query data with the same granular permissions as the person using it. Managing authentication, service keys, and tokens inside the product made framework dependence difficult to build and scale. Dougherty says abstractions can help during prototyping, but teams should consider their production requirements before making a third-party library central to the system.
A manager agent and a small worker team keep multi-agent work bounded
For multi-agent systems, Dougherty recommends a hierarchy with a manager agent responsible for the final outcome. Worker agents can receive narrower instructions, more context for their tasks, and more specific tools. He says teams should usually stay between five and eight cooperating agents. Systems with 25 or 50 agents often reduce the chance of completing the real task because they can enter loops or follow paths that never return. He also prefers describing and rewarding the overall objective, then letting the manager coordinate workers, rather than forcing every worker through a fixed sequence.
"If you could limit yourself to about between five and eight agents working together, then that was typically a task that could be accomplished well by a multi-agent team."16:06
Who should watch
You are building an agent that keeps writing plausible answers even when the available data cannot support them.
Your prototype works, but production will require user-specific permissions, tool-call control, and a decision about whether to adopt a framework.
You are splitting work across multiple agents and need practical limits on hierarchy size and coordination.
