Gemma 4 2B and 4B models bring tool calling, structured JSON output, reasoning, and multimodal skills to local devices.
2
LiteRT provides a cross-platform path for deploying TensorFlow, PyTorch, and JAX models across mobile, desktop, web, and IoT.
3
On-device inference can reduce latency, protect sensitive data, support offline use, and lower cloud costs, with hardware acceleration affecting the final performance.
Summary
Chintan Parikh presents Google's current approach to running AI on edge devices. He focuses on Gemma 4 2B and 4B models, which support local tool calling, structured JSON output, thinking mode, image understanding, and agent skills. Demos show local applications for Wikipedia queries, mood and sleep tracking, image-based music generation, and animal sound workflows. The talk then moves to LiteRT, Google's renamed and expanded on-device framework. It supports TensorFlow Lite models and conversion paths for PyTorch and JAX models, with deployments across Android, iOS, desktop, web, Raspberry Pi, and other IoT devices. Parikh explains the surrounding tools for conversion, quantization, graph inspection, benchmarking, and CPU, GPU, and NPU acceleration. He gives performance figures from Google's tests, while also acknowledging that device coverage and model performance need to be evaluated for each deployment.
On-device inference changes the tradeoff between speed, privacy, connectivity, and cost
Parikh frames edge inference around practical deployment needs. Real-time camera effects and video calls need low latency, which local processing can provide. Sensitive documents and summaries may be better kept on the device for privacy. Local models also continue to work when connectivity is poor. He describes hybrid designs where developers decide what should run locally and what should run in the cloud, partly to manage token and infrastructure costs. The choice depends on the application, since larger models need more memory and some workloads still benefit from remote services.
Gemma 4 edge models add agent features to small local models
The talk focuses on Gemma 4 2B and 4B models for on-device use. The 2B model uses roughly 1 to 2 GB of RAM after quantization and is suited to voice interfaces, summarization, and other local processing. The 4B model needs more memory and is aimed at larger platforms such as laptops and IoT devices. Both models add built-in function and tool calling, allowing local inference to interact with APIs. They also support structured JSON output through the model architecture, along with a thinking mode for reasoning.
The Gallery app turns local agent skills into working examples
Google's Gallery app provides a playground for trying the Gemma 4 edge capabilities. Its examples include audio scribe, image questions, Wikipedia lookup, mood and sleep tracking, image understanding, music generation, and workflows that move between applications. The app runs these experiences on the device and includes sample code and instructions for creating skills. Developers can fork the open-source sample app, download existing examples, and share their own skills through a GitHub repository. Parikh presents the app as a way to learn the model capabilities by building directly on the examples.
LiteRT extends the TensorFlow Lite deployment path across frameworks and platforms
LiteRT is Google's on-device framework, built on the TensorFlow Lite foundation and its model format. Existing TensorFlow Lite models continue to run on it. The rebranding also reflects support for bringing in PyTorch and JAX models by converting them to the supported format. Parikh describes deployments across Android, iOS, macOS, Linux, Windows, web, Raspberry Pi, and other IoT devices. For large language models, developers can use the LiteRT LLM path. Other models use the standard LiteRT path.
Conversion, quantization, and benchmarking determine whether a model can ship
The deployment stack includes LiteRT Torch for conversion, quantization options, and Model Explorer for inspecting a model graph. Model Explorer lets developers decide which parts of a graph to modify and how to use mixed precision or full quantization. AI Edge Portal provides cloud-based benchmarking for Android fleets. It helps teams compare deployment recipes, including ahead-of-time and just-in-time compilation, before testing across a broad range of devices. This matters when an application must run on phones that are several years old as well as current hardware.
Hardware acceleration affects both response time and energy use
CPU and GPU support provide broad coverage across platforms, while Google is adding NPU integrations. Parikh says LiteRT has completed integrations with Qualcomm and MediaTek and is working on more partner support. He connects NPU use to real-time AR, text-to-speech, camera, and AR/VR workloads. In the examples he cites, NPU acceleration can provide a 3 to 10 times performance improvement and reduce energy use. Developers can choose the accelerator and deployment approach based on the device and workload.
Google's published measurements still need to be read as device-specific evidence
Parikh points attendees to performance data published with the quantized models on Hugging Face. He mentions up to a 13 times boost in some accelerator cases, around 56 tokens per second in one set of results, and up to 35 times faster mobile performance than Llama in Google's comparisons. He says desktop performance is at parity in the cited comparison and reports 3 times performance on IoT. The talk also demonstrates a Raspberry Pi robot running inference on a CPU, while admitting that the newly assembled demo still needs performance work.
"If you are working with PyTorch models, you can take one of those models, convert it to the TFLite file format, and then go through this journey, and deploy."13:00
Who should watch
You are building mobile, desktop, web, or IoT applications and need one on-device model deployment path across several platforms.
Your application handles sensitive data, needs to work offline, or has latency requirements that make sending every request to a cloud service impractical.
You need to compare quantization, compilation, and accelerator choices across a mixed fleet of Android devices.
