# Local AI on your own hardware

A pack of 10 talks from the AI Engineer YouTube channel, in the order to watch them. 5h 39m of video.
Page: https://aietalks.com/packs/local-ai

A personal agent may need your messages, calendar and accounts, which makes sending every thought and action through a cloud provider a hard bargain. That trust problem comes before the practical question: which jobs are small and measurable enough to move onto your own hardware? Quantization and llamafile show how models fit and travel; Foundry Local turns them into a desktop product instead of a folder of weights. On phones, the choices become concrete: Android can share Gemini Nano across apps, an app can ship a tiny fine-tuned function caller, or an iPhone can run Gemma 4 through MLX. The extravagant version comes last: a frontier model spread across home hardware, where memory bandwidth and the split between prefill and decode set the limits.

## This pack is for you if

- You want an agent to use private files and accounts without sending them to a provider.
- You are deciding whether a laptop or phone can replace a cloud model for one product task.
- You have tried local models, but the choices around runtimes, compression and hardware got in the way.

## The talks, in order

### 1. Personal, Local, Private AI Agents

Soumith Chintala, Meta | 20:32 | AI Engineer Summit 2025
Video: https://www.youtube.com/watch?v=jMoAaZP_Kkw
Summary: https://aietalks.com/talks/personal-local-private-ai-agents.md

Why first: Chintala starts with the personal reason to run locally. A continuous agent needs broad access to private services, yet its powerful actions make a cloud provider harder to trust. Khalil's panel comes next because it turns that private need into a choice of models, harnesses and hardware.

### 2. State of the Union: Why Local, Why Now

Nader Khalil, NVIDIA & Joseph Nelson, Roboflow & Alex Cheema, EXO Labs & Ahmad Osman, Osmantic & Matthew Berman, Forward Future | 44:29 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=KB41dTlX1Uc
Summary: https://aietalks.com/talks/state-of-the-union-why-local-why-now.md

Why second: Chintala gives the personal case for local AI. Khalil's panel widens it into a system design: frontier models can plan while smaller local models do the routine work. Nabors follows by showing how to decide which tasks can actually move onto the device.

### 3. Frontier Results, On Device

RL Nabors, Arize | 30:52 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=fWXJM-J0ZB8
Summary: https://aietalks.com/talks/frontier-results-on-device.md

Why here: Khalil's panel proposes using different models for different jobs. Nabors supplies the test: prove the task with a large model, then choose the smallest model that clears a measured bar and let the harness catch simple errors. The compression panel next tackles what happens when the chosen model still does not fit.

### 4. Compression at the Edge

Chris Alexiuk, NVIDIA & Daniel Han, Unsloth & Asma Beevi, NVIDIA & Merve Noyan, Hugging Face & Parth Sareen, Ollama | 46:01 | AI Engineer World's Fair 2026
Video: https://www.youtube.com/watch?v=J4_jCrTxMkk
Summary: https://aietalks.com/talks/compression-at-the-edge.md

Why here: Nabors chooses the smallest model that can do the job. The compression panel offers another option: shrink a larger model without blindly trusting benchmark scores. Hood and Tunney follow because fitting the model into local memory is only half the problem; someone still has to make it easy to run.

### 5. Llamafile: Bringing AI to the Masses with Fast CPU Inference

Stephen Hood & Justine Tunney, Mozilla | 17:25 | AI Engineer World's Fair 2024
Video: https://www.youtube.com/watch?v=-mRi-B3t6fA
Summary: https://aietalks.com/talks/llamafile-bringing-ai-to-the-masses-with-fast-cpu-inference.md

Why here: the compression panel gets a model down to a usable size. Hood and Tunney package the weights and runtime into one executable that can run across operating systems, even on a CPU. Ning comes next with the step beyond a portable runtime: a complete desktop application that chooses and manages local models for the user.

### 6. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime/Olive

Emma Ning, Microsoft | 22:52 | AI Engineer World's Fair 2025
Video: https://www.youtube.com/watch?v=l614N5W60ls
Summary: https://aietalks.com/talks/foundry-local-cutting-edge-ai-experiences-on-device-with-onnx-runtime-olive.md

Why here: Hood and Tunney make one model executable portable. Ning turns local inference into a desktop product, with the same document app running on Windows and Mac and a service managing the models underneath. Muntenescu and Gaymond take that managed approach onto a phone, where the operating system shares one model across apps.

### 7. Gemini Nano on device

Florina Muntenescu & Oli Gaymond, Google DeepMind | 19:38 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=owH1f0N-keY
Summary: https://aietalks.com/talks/gemini-nano-on-device.md

Why here: Ning's desktop app relies on a local service to manage models. Muntenescu and Gaymond show the phone version: Android installs Gemini Nano once, schedules its work and lets several apps call it. Brick follows with the alternative, where the app ships its own smaller model and accepts more setup in return for more control.

### 8. From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents

Cormac Brick, Google | 21:01 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=-TiET_K-E_g
Summary: https://aietalks.com/talks/from-46-to-90-fine-tuning-tiny-llms-for-on-device-agents.md

Why here: Muntenescu and Gaymond make the case for a system model that apps share. Brick puts that beside an app-level model tuned for one narrow job, including a tiny function caller lifted from 46 percent to over 90 percent on most app intents. Grondin comes next with the corresponding route for developers building on Apple hardware.

### 9. Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX

Adrien Grondin, Locally AI | 10:51 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=a2muGkT4WD4
Summary: https://aietalks.com/talks/running-llms-on-your-iphone-40-tok-s-gemma-4-with-mlx.md

Why here: Brick shows why an app might ship and tune its own phone model. Grondin gives Apple developers a short route from MLX Swift LM to a working iPhone app, then shows Gemma 4 running at about 40 tokens per second. Cheema closes the pack by pushing past phone-sized models to the outer limit of hardware a person might keep at home.

### 10. Run Frontier AI at Home

Alex Cheema, EXO Labs | 1:45:02 | AI Engineer Europe 2026
Video: https://www.youtube.com/watch?v=ESbWpPT_9-o
Summary: https://aietalks.com/talks/run-frontier-ai-at-home.md

Why last: Grondin shows what one current iPhone can do. Cheema asks what it takes to run a frontier model at home, where decode becomes a memory-bandwidth problem and different devices can split prefill from decode. It belongs last because the private personal need from Chintala has now become a demanding hardware and systems project.