# Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime/Olive

Emma Ning, Microsoft | AI Engineer World's Fair 2025 | 22:52

Source: https://www.youtube.com/watch?v=l614N5W60ls
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/foundry-local-cutting-edge-ai-experiences-on-device-with-onnx-runtime-olive
Published: 2025-06-27
Tags: edge, inference, mcp, privacy

## TL;DR
- Local AI avoids cloud dependency when applications need offline access, privacy, lower inference costs, or real-time responses.
- Foundry Local combines ONNX Runtime, a local model management service, a CLI, and SDKs for cross-platform applications on Windows and macOS.
- Developers can run local agents built from a model and MCP servers, with permissions controlling access to files and other tools.

## Summary
Emma Ning presents Foundry Local as a way to build AI applications that run directly on Windows and macOS devices. She gives four reasons to use local inference: unreliable or unavailable network access, sensitive data that must stay on the device, the cost of large numbers of cloud inference calls, and applications that need real-time responses. Foundry Local uses ONNX Runtime to accelerate models across different hardware, while its management service downloads and manages models locally. The CLI lets developers list models, inspect cached models, and run them. Ning then demonstrates a cross-platform document summarizer built with the JavaScript SDK, using the same project on Windows and Mac. Her final demo uses a local agent with a language model, a file-system MCP server, and an OCR MCP server to find a receipt and extract its total. She is honest that local models are generally less capable than cloud models, so developers must choose tasks that fit their limits.

## Key ideas
### Local AI handles constraints that cloud inference cannot
[00:41](https://www.youtube.com/watch?v=l614N5W60ls&t=41s)
Ning gives four reasons developers may need local inference. Applications must continue working with low bandwidth or no network access. Sensitive information, such as legal documents and patient data, may need to stay entirely on the device. High-volume applications, including games deployed to millions of devices, can make hundreds of millions of inference calls each day, which creates a cost problem for cloud services. Applications that need real-time responses also cannot always wait for a round trip to the cloud.

### Hardware and smaller models have made local inference practical
[02:08](https://www.youtube.com/watch?v=l614N5W60ls&t=128s)
Ning says modern client devices increasingly include GPUs and MPUs that can run advanced AI models. Model companies are also publishing leaner and faster models designed for local inference. She names Phi-4-mini and DeepSeek variants as examples, alongside runtime-level optimization techniques. In her view, progress in hardware, model design, and runtime optimization has converged enough to make local AI a practical option.

### Foundry Local packages Microsoft's existing AI infrastructure for device use
[03:07](https://www.youtube.com/watch?v=l614N5W60ls&t=187s)
Foundry Local brings together Azure AI Foundry, ONNX Runtime, and Windows. ONNX Runtime provides the cross-platform inference engine and accelerates models across different hardware. A new management service hosts and manages models on the client device, while connecting to Azure AI Foundry to download open-source models when needed. The Foundry Local CLI helps developers explore models, and SDKs let applications use the service across cloud and local environments.

### The CLI exposes model variants for different hardware
[09:04](https://www.youtube.com/watch?v=l614N5W60ls&t=544s)
In the CLI demo, Ning installs Foundry Local with winget on Windows and Homebrew on macOS. The model list command shows supported generative AI models and variants optimized for CPU, CUDA, integrated GPU, and MPU hardware. The cache command shows models already downloaded to the device. Developers can compare models during experimentation before selecting one for an application. Ning runs a 1.5-billion-parameter model and reports around 90 tokens per second in the latency test.

### The SDK supports the same local application on Windows and Mac
[13:09](https://www.youtube.com/watch?v=l614N5W60ls&t=789s)
Ning builds a document summarization application for an internal project whose documents cannot be uploaded to the cloud. The app accepts a URL or local file, lets the user choose a model, and produces a shorter project overview. The demo uses the JavaScript SDK to create and initialize a Foundry Local manager with a model name, then sends requests through the local endpoint. Ning packages the project for a colleague on Mac, who runs the same code and interface and chooses a different model.

### A Foundry Local agent combines one model with MCP servers
[18:08](https://www.youtube.com/watch?v=l614N5W60ls&t=1088s)
Ning describes an agent as one model plus one or more MCP servers. The agent feature is in private preview during the talk. In the demo, an OCR agent uses Phi-4-mini, a file-system MCP server, and an OCR MCP server. The CLI shows the agent's dependencies and tools, then asks for permission to access the selected directory and OCR service. The available tools include file-system operations and OCR functions.

### Permissions and tool selection are part of the local agent flow
[20:19](https://www.youtube.com/watch?v=l614N5W60ls&t=1219s)
The demo asks the agent to find a receipt in a local folder, process it, and return the total amount. The agent first chooses a file-search tool to locate the receipt. It then uses OCR to extract the text and obtains the amount. Foundry Local asks for permission before using the MCP servers and the directory, so the agent's access is explicit rather than unrestricted.

### Local models require narrower expectations than cloud models
[21:57](https://www.youtube.com/watch?v=l614N5W60ls&t=1317s)
Ning closes by saying that local models are generally less capable than cloud models. Developers should not expect a local model or agent to perform the same fancy tasks as a cloud model. She still sees substantial potential in local applications, especially when privacy, offline operation, cost, or response time matters. Her examples show a practical way to select a smaller model and pair it with local tools for a defined task.

## Notable quotes
- "So that's why we need a local AI." (02:08)
- "We are not starting from scratch." (04:30)
- "Foundry Local enables you to easily create and build and run a local agent using local model and MCP servers." (18:08)
- "You cannot expect it to do the fancy work that cloud model or cloud agent can do." (22:09)

## Tools & references mentioned
- Foundry Local
- ONNX Runtime
- Olive
- Azure AI Foundry
- Windows
- macOS
- Microsoft
- NVIDIA
- Intel
- AMD
- Qualcomm
- Phi-4-mini
- DeepSeek
- Pieces
- Llama C++
- Ollama
- MCP

## Who should watch
- You are building an application that must keep sensitive documents on the device or continue working without dependable network access.
- Your team needs one local inference path across Windows and Mac, with model selection and hardware-specific variants exposed through a CLI or SDK.
- You are evaluating local agents and need a concrete example of combining a small model with file-system and OCR MCP servers.

## Related talks

- [Run Frontier AI at Home](https://aietalks.com/talks/run-frontier-ai-at-home) (Alex Cheema, EXO Labs, 1:45:02)
- [The Desktop Frontier](https://aietalks.com/talks/the-desktop-frontier) (Ahmad Osman, Osmantic, 18:02)
- [Llamafile: Bringing AI to the Masses with Fast CPU Inference](https://aietalks.com/talks/llamafile-bringing-ai-to-the-masses-with-fast-cpu-inference) (Stephen Hood & Justine Tunney, Mozilla, 17:25)
- [Why MLX](https://aietalks.com/talks/why-mlx) (Prince Canuma, Neywa Labs, 23:10)
- [State of the Union: Why Local, Why Now](https://aietalks.com/talks/state-of-the-union-why-local-why-now) (Nader Khalil, NVIDIA & Joseph Nelson, Roboflow & Alex Cheema, EXO Labs & Ahmad Osman, Osmantic & Matthew Berman, Forward Future, 44:29)