On-device AI gives developers lower latency, privacy, offline use, and control over whether the model is provided by the operating system or shipped inside the app.
2
Gemini Nano through AI Core is the simpler system-level option, while LiteRT-LM lets developers package custom models with an app for narrower tasks.
3
Function Gemma improved from 46% out-of-the-box accuracy to over 90% for eight of ten app functions after fine-tuning on synthetic data.
Summary
Cormac Brick presents two ways to build on-device AI. Developers can use Gemini Nano through Android's AI Core when a preinstalled model meets their needs, or ship a custom model with the app using LiteRT-LM. He demonstrates a skill harness that gives Gemma 4 access to skills on demand, with JavaScript rendering for interfaces such as a restaurant roulette wheel. The second half focuses on tiny models below one billion parameters. Brick explains the export path from Transformers and PyTorch-based models into LiteRT-LM, then describes Function Gemma, a 270-million-parameter model for function calling. Fine-tuning with synthetically generated examples raised an app-intent test from 46% to over 90% for eight of ten functions. He also describes Eloquent, an offline transcription app that chains small speech recognition and text-polishing models. The talk is practical about the tradeoff: tiny models can work well, but they need focused tasks and more preparation than prompting a larger model.
On-device AI has practical reasons beyond model size
Brick gives latency, privacy, offline use, reliability, and possible cost savings as reasons to run AI locally. He describes the Google AI Edge stack, including MediaPipe, LiteRT-LM, and the LiteRT runtime, formerly known as TensorFlow Lite. LiteRT can run models on a CPU, GPU, or NPU, depending on the platform and the developer's choice. A version is built into Android OS, and Brick says the runtime supports over 2.7 billion devices. This makes the talk's focus broader than a single phone demo. The same deployment stack is intended to work across several platforms, with Gemma models available on many of them.
System-level and app-level AI give developers different tradeoffs
Brick separates system-level GenAI from app-level GenAI. Gemini Nano through AI Core is preinstalled and optimized for the device, so an app can use it without increasing its download size. He gives summarization as an example. This is the best starting point when the available model meets the product's needs. LiteRT-LM is the alternative for a more specific or customized task. The model can ship with the app or be loaded by a web page, which gives the developer more control and access to smaller models across more devices. Brick is direct that this option takes more work.
The skill harness loads only the skill a request needs
The skill harness puts system prompts and skill descriptions into the model's context, while keeping the full function details out until they are needed. If a user asks for the location of a Google office, the model chooses a map skill, calls a load-skill tool, and then uses a show-JavaScript tool to display the result in the app. Skills can include JavaScript for custom interfaces. Brick points to the restaurant roulette demo, whose wheel is rendered with custom JavaScript. The approach lets one app expose several capabilities while keeping each skill's detailed instructions on demand.
Developers can write, test, and publish skills from the gallery
Google AI Edge Gallery includes instructions for writing skills by hand and lets developers load a published skill from a URL. Brick says developers can also use skills to write skills, with Gemini CLI and Code Coach helping his team create about 80 examples. The team uses an ADB skill to test and debug against a device. A developer can publish a skill in a GitHub repository, load it through the app's menu, and share it with the community through the project's GitHub discussion. Brick says the feature had only been public for about a week, so the examples were still early community work.
LiteRT-LM packages a deployable model into one file
LiteRT-LM runs models in a LiteRT-LM format packaged as a single file containing what the runtime needs to execute the model. Brick describes it as open source, fast, and cross-platform. At the time of the talk, C++ and Java APIs were available, while Swift and JavaScript APIs were still coming. The export flow starts with models from Transformers, uses LiteRT Torch to help export them, and then runs them with LiteRT-LM on a desktop or in the Gallery app before deployment. He also shows a 500-million-parameter visual language model running with Qualcomm NPU acceleration.
Tiny models need narrow tasks and often need fine-tuning
Brick says models around 100 or 200 million parameters usually need a narrow, focused task to work well. Developers can start with prebuilt models for jobs such as visual language processing or transcription, but fine-tuning is often needed for a particular app. His recommended workflow uses synthetic data. Function Gemma is a 270-million-parameter model built for function calling. It can handle text-to-function or voice-to-function use cases, and Brick says a Pixel 7 can process almost 2,000 tokens per second during prefill and 140 tokens per second during decode. The model is small enough to remain useful on older devices.
Synthetic fine-tuning raised Function Gemma's app-intent accuracy
For an app-intent test covering functions such as adding a calendar event or email, Brick says Function Gemma reached about 46% out of the box. The team then generated a synthetic dataset with Gemini Flash based on its app functions and used that data in a fine-tuning workflow. Accuracy rose to over 90% for eight of the ten functions, while two functions remained lower. Brick contrasts this with a larger model, where the functions could be supplied through a system prompt. The tiny model needs the examples built into its training process, which takes more effort but can produce a model suitable for shipping in an app.
Eloquent chains two small models for offline transcription
Brick uses Eloquent as a production example of the small-model approach. The app transcribes speech and supports a personal dictionary for technical terms and names. Under the hood, it chains an automatic speech recognition engine based on Gemma 3 technology with a separate text-polishing engine, also based on Gemma 3 technology. Each model has only a few hundred million parameters. Together they produce offline transcription that can use personal vocabulary and remove filler sounds such as ums and ahs. Brick presents Eloquent as evidence that small models can support a useful product after focused model work, while also noting that the app was not yet widely available.
Multiple skills across a conversation work better than one multi-skill turn
In the question period, Brick says the four-billion-parameter model can choose reasonably well among about eight enabled skills. A conversation can use one skill and then another, such as retrieving a fact from Wikipedia and showing its location on Google Maps. The harder case is a single interaction where the app must call several skills as part of one answer. Brick says that works sometimes and remains an area the team is still testing. He describes the harness itself as simple and says the team is still discovering how far the model can be pushed.
"They're small enough to build into your app if you want to have more customization or you want to do something that isn't already available for you in AI Core."00:32
Who should watch
You are choosing between a preinstalled mobile model and a custom model that ships inside your app.
You need function calling for a narrow on-device task and want a concrete fine-tuning workflow using synthetic data.
You are building offline transcription or another product that can combine several small, focused models.
