Running models on phones can reduce latency, preserve privacy, support offline use, and avoid sending every request to a cloud service.
2
Gemma 4's E2B and E4B models support function calling and thinking, which lets on-device agents load and use skills dynamically.
3
Tiny models below one billion parameters can support widely deployed app features when developers fine-tune them for narrow tasks and quantize them for edge hardware.
Summary
Cormac Brick explains how Google's AI Edge stack brings language models to phones, laptops, IoT devices, and other constrained hardware. The talk separates system-level AI, where an operating system provides a shared model, from in-app AI, where an app ships a task-specific model. Gemma 4's E2B and E4B models can run across mobile and embedded platforms and support the function calling needed for on-device agent skills. Brick describes a skill architecture that loads short descriptions first and fetches detailed instructions only when needed. He then turns to tiny models, showing how fine-tuning, quantization, and hardware-specific compilation can make narrow features practical across a wide range of devices. The AI Edge Eloquent app illustrates this approach with separate speech recognition and text-polishing models. The talk is practical about constraints: memory, context size, safety boundaries, platform support, and the tradeoff between separate models and one larger model.
Edge inference improves latency, privacy, offline access, and operating cost
Brick gives four reasons to run AI on the device. Low latency matters for experiences such as live voice translation, which Google shipped on Pixel and which is difficult to deliver through a cloud service at the required speed. Privacy matters when users want messages to remain encrypted on their phone. Offline operation removes the need for a network connection. Brick also sees laptop users experimenting with small language models in desktop agent workflows to reduce cloud costs. These benefits apply differently by use case, but they explain why mobile and edge deployment remains attractive even as cloud models grow more capable.
LiteRT and LiteRT-LM package one deployment path across many device classes
Google's stack includes MediaPipe for app development, LiteRT for standard inference, and LiteRT-LM for language-model runtimes. Brick says a LiteRT file can run on Android, iOS, macOS, Linux, Windows, the web, and IoT devices, using CPU or GPU execution. NPUs require extra compilation and produce a device-specific NPU file. LiteRT-LM packages the model, tokenizer, and other pieces needed for the autoregressive loop. The runtime exposes cross-platform APIs and can use hardware acceleration, so the application workflow stays similar even when the build path differs for CPU, GPU, and NPU.
System-level AI and in-app AI solve different deployment problems
Brick describes system-level generative AI as a shared model integrated into the operating system. Android AI Core and Apple's comparable system provide APIs such as summarization and prompting, with the model preloaded on selected devices. Developers customize this model through prompts or skills. In-app generative AI is shipped with an app or web page and targets wider reach, including less powerful devices. For narrow tasks such as summarization, transcription, and voice-to-action, Brick says fine-tuned models in the 100 to 500 million parameter range can provide reliable performance. These task-specific models are more suitable when an app cannot assume a premium phone.
Gemma 4 E2B and E4B make on-device skills possible through function calling and thinking
Gemma 4's two smaller models are E2B and E4B. The names refer to about two billion and four billion parameters that need to remain resident in RAM, while other parameters are memory-mapped and loaded selectively during inference. Brick says both models support audio, images, and text, and are on the AI Core roadmap. Their built-in function calling and thinking support allows an app to describe a skill and let the model choose and use it. The models can run across mobile, desktop, and embedded platforms, although the larger E4B model needs more computation and memory.
Progressive disclosure keeps an edge agent's context smaller and more reliable
The skill system exposes only a one-line description of each skill at first. When the model decides that a skill is relevant, it calls a load-skill function and receives the full skill.md instructions, including available function calls. It can then run JavaScript or another supported action. Brick uses this progressive disclosure pattern to avoid placing every tool's full schema in the context window. He says that long contexts are especially difficult for lighter models, so a more condensed context can improve the chance of producing a useful result. Skills can extend inputs through services such as Wikipedia or weather and outputs through maps, cards, or other interfaces.
Constrained decoding adds stronger tool-call guardrails for small models
LiteRT-LM applies constrained decoding when the model is generating a tool call. The constraint can be limited to the finite set of tools available to the application, rather than applying only generic JSON rules. Brick says this produces a more reliable overall system and is especially helpful for the two-billion-parameter model. Larger models need this strict constraint less often, but small on-device models benefit from the tighter output space. The skill architecture also supports native Android intents, local JavaScript, web APIs, and user-provided API keys, so the safety and reliability boundary includes both the model's output and the actions it is allowed to invoke.
Community skills and Gemini CLI lower the cost of prototyping on-device agents
AI Edge Gallery lets users toggle skills, load a skill from a URL, and add API keys for services that require them. Google publishes source code and hosts a GitHub discussion where users share skills, with selected community contributions becoming featured skills in the app. Brick says his team built about 80 internal skills, and more than half initially used Gemini CLI, Claude Code, or Antigravity to generate them. Gemini CLI could also use an Android ADB skill to test an app on a connected phone and iterate after basic validation. This makes skill development accessible without requiring every behavior to be built into the model.
Tiny models need task-specific fine-tuning for production reliability
Brick defines tiny models as models with fewer than one billion parameters. They are useful for shipping language features inside apps because they can reach more devices than larger system models. For models below roughly 500 million parameters, he recommends fine-tuning when developers need production-level reliability for their own task. Function Gemma, a 270-million-parameter model, was evaluated on voice-to-function calling across 10 Android-relevant functions and reached about 85 to 90 percent reliability in an internal evaluation. Brick says fine-tuning often improves a tiny-model evaluation by 20 to 40 points. Quantization then helps make the resulting model small enough to distribute widely.
Separate speech recognition and text polishing models make a practical offline app
AI Edge Eloquent uses a speech recognition engine to produce an unfiltered transcription, then sends that text to a separate small language model for polishing. The polishing stage removes interjections such as 'um' and 'ah', cleans up spoken corrections, and uses a biasing dictionary for names or technical terms. Users can add terms themselves, or the app can find unusual words through connected account data. Brick says the models are derived from the smaller Gemma 3 lineage and fine-tuned separately for transcription and polishing. A single model might combine the jobs, but separate models are easier to reuse, inspect, and debug across different app features.
"If we had to load all of the details for all of the skills into the edge model that would be a lot of context for the model to reason over."24:34
Who should watch
You are building an Android, iOS, laptop, or IoT app that needs local inference and want to compare system-provided models with models shipped inside the app.
You need an on-device agent that can call tools, load skills conditionally, or render richer outputs through JavaScript and native intents.
You are choosing a tiny model for transcription, function calling, text cleanup, or another narrow feature and need a concrete fine-tuning and deployment workflow.
