Loading every tool schema into every request makes agents slower, more expensive, and less accurate as the catalog grows.
2
A semantic tool router retrieves a small set of relevant tools and injects only those schemas into the model call.
3
Teams should start with a router around five retrieved tools, then evaluate accuracy, latency, failures, and fallback use on real requests.
Summary
Sohail Shaikh and Ankush Rastogi describe why the common fat-agent design degrades as an agent gains tools. Every request carries every function name, description, and JSON schema, even when most tools are unrelated. Their benchmark shows accuracy falling from about 78% with 10 tools to about 13% with 1,041 tools, while time to first token also rises. Their alternative is semantic routing, which indexes tool descriptions, retrieves the most relevant tools for each query, and injects only those schemas into the model prompt. They recommend starting with K equal to five, while testing K values of three, five, and ten. The speakers also cover implementation, logging, weak descriptions, rare tools, router misses, and fallback retrieval. They are clear that routing is unnecessary for small catalogs. The approach becomes useful when prompt size, latency, or tool confusion creates a production problem.
A large tool catalog makes the agent choose from too much irrelevant context
The fat-agent design sends every tool definition with every request. That includes function names, descriptions, and JSON schemas for actions the user may not need. The speakers say this can work in a demo or with around 10 tools, but problems appear as the product grows. At 30 or more tools, the model may confuse similar functions, invent tool names, or take longer to respond. Their example has 741 tools requiring up to 127,000 tokens before the user's question is even considered. They describe this as context overload. The problem comes from forcing every request to carry the entire catalog, rather than from one badly written tool.
Fat-agent tool selection accuracy falls sharply as the catalog grows
The benchmark comparison shows the effect of catalog size on tool selection. With 10 tools, the fat agent chooses correctly about 78% of the time. At almost 100 tools, accuracy falls to around 40%. With 741 tools, it reaches only 13.6%, which the speakers describe as roughly one correct tool out of eight. They connect this partly to the lost-in-the-middle problem: models pay stronger attention to the beginning and end of long context, while hundreds of schemas packed into the middle are less reliably used. The semantic router stays above 83% across the catalog sizes shown because the model chooses from a small, relevant set.
Large tool prompts add both token cost and first-token latency
Every request carrying 741 tool schemas pays for roughly 127,000 tokens of descriptions and schema text. The speakers point out that at 100,000 requests per day, this becomes billions of tokens spent only on describing tools. Just-in-time routing can include three to five relevant schemas, closer to about 1,000 tokens, which they describe as roughly a 99% reduction in tool-context tokens. Latency also grows with the fat-agent path because the model must process the larger prompt before answering. With 500 tools, first-token latency can pass five seconds. The router keeps the working prompt smaller and makes response time more predictable.
Semantic routing retrieves tools before the model sees their schemas
The routing pattern follows a retrieval workflow. Each tool gets a clear description, such as searching flights, checking calendar availability, or retrieving an order status. Those descriptions are embedded offline and stored in a vector index. At runtime, the user's query is embedded with the same model, and the router searches for the closest tool descriptions. It returns the top K tools, often three or five, and only those schemas are injected into the model call. The speakers compare this with RAG for tools. Teams that already have an embedding model and vector database can reuse much of their existing retrieval infrastructure.
Just-in-time context injection keeps the model's working set small
Ankush Rastogi separates semantic routing from just-in-time context injection. Semantic routing is the retrieval layer, while just-in-time injection is the context-management strategy. The fat-agent design loads everything before understanding the query. The just-in-time design waits until the query is known, then adds only the context needed for that request. The speakers relate this to lazy loading, just-in-time compilation, and on-demand resource loading. They cite an Anthropic report on on-demand tooling through MCP, where token usage fell from 150,000 to 2,000, a 98.7% reduction. Their conclusion is that a large catalog should be retrieved when needed rather than dumped into every prompt.
The router's K value controls the speed and coverage trade-off
The evaluation measures tool-selection accuracy, time to first token, input tokens per request, and estimated cost per thousand calls. The speakers test tool pools with 10, 50, 100, 200, and 1,041 tools, using the same queries, model, answer key, and catalog in both modes. They also vary K, the number of tools passed to the model, across 3, 5, and 10. Smaller K is faster and cheaper. Larger K can recover more edge cases. They recommend K equal to five as a practical starting point, followed by evaluation against the team's accuracy target rather than treating that value as universal.
A production router needs an indexed catalog, logging, and evaluation
The implementation has three operational steps. First, store each tool's name, description, and schema, embed the descriptions, and save the vectors in a database such as ChromaDB, Pinecone, or Qdrant. For each request, embed the user query and run nearest-neighbor search to retrieve the top K tools. Then fetch the selected schemas, pass only those schemas to the model, and log the selected tool. Logging lets a team improve descriptions or tune K when a tool is missed. The speakers describe the runtime overhead as one embedding call and one vector search. They also recommend re-embedding tools when descriptions or schemas change.
Routing can remove incorrect tools from the model's choice set
The speakers illustrate routing with a 200-tool agent. For a request to find a flight to New York next Wednesday, the router might return search flights, book flight, and calendar check. Unrelated hotel, weather, email, SQL, and workflow tools are left out of the model call. For a weather request about Paris, the router returns weather and forecast tools, while flight and hotel tools are not injected. This does more than add likely candidates. It removes unrelated functions that could compete for the model's attention. The speakers present this reduction in the choice set as a practical way to reduce cross-tool confusion.
Router misses and weak descriptions need explicit recovery paths
The speakers identify several risks. A router can miss the tool the model needs, so the system can widen K, run a second retrieval pass, or route to a broader tool group through a fallback. Weak descriptions also produce weak embeddings. Descriptions should use the words users actually say and include the intent, action, and important entities. Rare tools may never rank highly unless their descriptions contain the right language, so teams should monitor misses and rewrite those descriptions. They advise against routing very small systems. Static loading may still be fine with 10 or 15 tools, while routing becomes more useful when prompt size, latency, or tool confusion causes a real production issue.
"The design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog."03:05
Who should watch
You have an agent that sends dozens or hundreds of tool schemas with every request and want to reduce prompt size or first-token latency.
Your model increasingly confuses similar functions as the tool catalog grows, and you need a retrieval and fallback design.
You already run embeddings and a vector database for RAG and want a concrete way to reuse that setup for tool selection.
