AI systems draw on intrinsic knowledge in model parameters, extrinsic knowledge from organizational data, and learned knowledge from how agents perform tasks.
2
Company agents need grounding across documents, email, chats, analytics assets, private data, and the web, with retrieval systems that combine several search methods.
3
Foundry Agent Optimizer evaluates an agent, generates candidate configurations, and applies improved instructions based on evaluation results and agent traces.
Summary
Pablo Castro presents AI applications through three kinds of knowledge. Intrinsic knowledge is the information stored in a model's parametric memory, which powered early tools such as IntelliSense, GitHub Copilot, and ChatGPT. Extrinsic knowledge comes from the data an agent needs while working inside an organization. Castro describes Microsoft IQ capabilities for work data, analytics data, private agent data, and web information. He also explains how retrieval has moved beyond vector search toward layered systems that combine methods and can use agentic retrieval for harder information needs. In a Foundry demonstration, he creates a knowledge base from PDFs, parquet tables, and web data, then connects it to an agent. Learned knowledge comes from observing agent work and improving its configuration. The Agent Optimizer evaluates a baseline, generates candidates, and applies a better configuration based on task performance and traces.
AI applications begin with knowledge stored in model parameters
Castro calls intrinsic knowledge the information that comes with a model. It comes from training data and is stored in the model's parametric memory. He argues that this knowledge helped create the rapid growth around current AI applications. He compares writing code with and without these systems, then traces coding assistance from Microsoft's 1996 IntelliSense, which removed the need to remember function signatures, to machine-learning ranking, GitHub Copilot, and later coding tools. GitHub Copilot and ChatGPT relied heavily on what models already knew, along with their ability to reason.
Agents working inside companies need access to ambient organizational data
Model knowledge alone is insufficient when an agent must participate in company work. Castro describes the data surrounding an organization, including documents, email, chat threads, data warehouses, and other information that an agent may need after it leaves its narrowly managed domain. Microsoft IQ provides entry points into these sources. Work IQ connects to documents, email, calendars, chats, and relationships between people. Fabric IQ covers analytics assets such as warehouses, lakes, and Power BI reports. Foundry IQ handles data supplied for agents, while Web IQ provides public web information.
Retrieval quality improves when systems combine search methods
Castro describes the move from simple vector databases to more complicated retrieval systems. Vector search made it easier to get retrieval applications running, but he says cosine similarity alone was not enough. An evaluation from the search technology behind Foundry IQ showed that combined methods performed better than individual methods in real customer scenarios. Foundry IQ therefore layers its system. A user can provide documents and let the platform handle chunking, vectorization, relevance, ranking, and agentic retrieval, or an expert can control vector indexes, quantization, and lexical retrieval within the same stack.
Agentic retrieval checks whether the retrieved information answers the need
For simple cases, Castro says single-shot retrieval can be sufficient. More difficult cases benefit from a retrieval system that examines the data, reflects on the information need, and decides whether the gathered material is enough before returning results. In Microsoft's evaluations, he says agentic retrieval did better on measures such as evidence recall and answer completeness than the simpler individual methods. The system is designed to expose control when users need it while keeping routine cases easier to configure.
A Foundry knowledge base can combine unstructured, structured, and web data
In the Foundry demonstration, Castro creates a knowledge base for a movie dataset. He gives the agentic retrieval workflow a model and selects an effort level that trades latency against quality. He then connects PDFs and other unstructured files in blob storage, parquet tables containing statistics, and web data. The resulting knowledge base can connect directly to a Foundry agent. Castro says every knowledge base is also an MCP server, so an existing harness can connect to it without extra glue code. Users who need deeper control can inspect indexes, chunk organization, vector settings, and indexing algorithms.
Learned knowledge comes from observing work and improving agent behavior
Castro defines learned knowledge as the result of work done by people and organizations every day. Agents make it possible to observe those processes, evaluate what happened, and improve the steps. He cites Satya Nadella's writing about people and agents compounding how they work. In this view, an organization can capture practices that are specific to it and use them to change how its agents operate. The Agent Optimizer is Microsoft's practical version of this idea. It evaluates a baseline, generates candidates, measures them, and can deploy a better result.
Agent optimization depends on externalized configuration and evaluation
The optimization demonstration uses an agent in VS Code with the Foundry toolkit. The agent's instructions, tool definitions, skills, and other configuration are kept outside the code. If an evaluation dataset does not exist, the toolkit can generate one from agent traces and instructions, with a focus on task adherence. The optimize command evaluates candidates and uses a hill-climbing loop to improve the selected metric. Once a better candidate is found, optimize apply swaps the baseline configuration for the new one. The resulting instructions were generated through the process and were informed by agent traces.
"For more sophisticated cases, you do want a system that can reflect on what's in the data set and decide whether or not we've satisfied the information need as stated in the input before we come back with results."09:04
Who should watch
You are building an agent that needs private company information in addition to its model's built-in knowledge.
Your retrieval stack relies mainly on vector search and you need to understand when combined or agentic retrieval may help.
You want to improve an existing agent through evaluations, generated test cases, and configuration changes rather than hand-editing every instruction.
