RAM cost is the main limit for running AI on edge devices, so model size and quantization determine which products can use local inference.
2
Small models of 1 to 4 billion parameters can run on phones, laptops, Raspberry Pis, and robotics hardware, while tiny models reach devices with less than 2 GB of RAM.
3
Fine-tuning tiny models with synthetic data makes focused features such as voice-to-function calling and offline dictation practical.
Summary
Cormac Brick explains why edge AI needs models that fit the device's RAM budget. Cloud inference brings latency, connectivity, privacy, and cost problems, while DRAM is becoming more expensive and some phones are shipping with less of it. Quantized 2 billion parameter Gemma models can run on devices such as Raspberry Pi, Jetson Orin Nano, and Qualcomm hardware, but they still require several gigabytes of RAM. Tiny models, ranging from about 50 million to 500 million parameters, reach older laptops, cheaper devices, and applications where AI is only a small background feature. These models often need fine-tuning rather than simple prompting. Brick shows how synthetic training data can produce reliable function calling, voice-controlled device actions, and offline dictation using two fine-tuned Gemma models. The talk is practical about the tradeoff: smaller models broaden deployment, but they require more task-specific preparation.
Edge inference avoids cloud latency, connectivity, privacy, and usage costs
Brick gives several reasons to run AI locally. On-device inference can provide fast, consistent responses, keep data on the device, and continue working without reception. It also avoids multiplying cloud costs across a large number of users and interactions. This matters for mobile apps and browser features, where even a low per-token price can add up at scale. The device must still have enough memory and compute, but local execution makes the feature available when a network connection is unreliable or absent. Brick frames edge deployment as a product decision as well as a performance decision.
Brick says the most significant current constraint is DRAM cost. Some phone manufacturers are putting less DRAM into devices than before, while the cost of a 6 GB Raspberry Pi has risen by about 2.5 times since launch. That makes memory footprint a first-order design problem. His team's response is to use quantization and find the smallest model that can perform a task. The wider device pool and the fact that much LLM research focuses on large models add further difficulties. Edge teams have to target hardware with very different memory and acceleration capabilities.
Two-billion-parameter models fit better through aggressive quantization
Brick describes small models as roughly 1 to 4 billion parameters. A 2 billion parameter Gemma model can use a mixture of 2-bit, 4-bit, and 8-bit quantization, reaching about 2.9 bits per weight for the weights held in memory. Per-layer embeddings are another part of the memory reduction work. In his example, the weights take about 841 MB, while the runtime and KV cache can bring active RAM use to around 2 GB. Once the operating system and other processes are included, he gives a practical rule of thumb of at least 4 GB for the device.
The same small model has very different speeds across edge hardware
Using the team's runtime, the quantized 2 billion parameter model decodes at about 7.6 tokens per second on a Raspberry Pi. A Jetson Orin Nano reaches about 24 tokens per second with that toolchain, and a Qualcomm IoT board reaches about 31 tokens per second. The Qualcomm board also gets close to 4,000 tokens per second during prefilling. Brick notes that a medium-resolution image can consume about 500 tokens and a high-resolution image about 1,120 tokens, so the NPU can support roughly a few high-resolution frames per second. More capable hardware makes these applications possible, but raises the device bill of materials.
AI Edge Gallery and a hobby robot make small-model deployment testable
Google AI Edge Gallery lets developers run supported small models on iOS and Android phones. The application is open source, so it also shows how to build a similar experience with the open source runtime. Brick recommends it as a way to measure model speed directly on a phone. He also describes an open-source mini robot built as a hobby project by an engineer at DeepMind. Versions using a Jetson Orin Nano and a Raspberry Pi can read signs, react, nod, and accept voice and image input. The Jetson version provides good real-time interaction, while the Raspberry Pi version is noticeably slower.
Tiny models reach devices where a small LLM is still too large
Some applications cannot afford a 1 to 4 billion parameter model. The AI feature may be only a small part of a larger application, or it may need to run alongside many other processes. Older laptops, cheaper consumer devices, and entry-level robotics hardware also have tighter limits. Brick describes tiny models ranging from about 50 million to 500 million parameters. They can require less than 2 GB of RAM and can run very quickly. The tradeoff is a harder deployment process. Prompting may work for larger small models, while tiny models often need a fixed-task model or fine-tuning.
Off-the-shelf tiny models already cover speech, vision, and embeddings
Brick says automatic speech recognition, vision, and embedding models are useful fixed-task options. He shows Apple FastVLM, a 0.5 billion parameter model running on an Android device with hardware acceleration, as an example of fast visual intelligence for an edge or IoT device. Speech models can provide transcription, while embedding models can process and match text. These models do not need to solve every general-purpose language task. Their narrower scope helps them fit devices that cannot host a general small LLM and makes them suitable for adding one specific capability to an existing product.
Synthetic data and fine-tuning turn tiny models into function callers
For a task such as converting free text into device actions, Brick recommends selecting a base model, checking its speed and memory footprint, then fine-tuning it. FunctionGemma has extra pre-training for function-calling patterns. In the mobile actions example, a fine-tuned model handles ten output functions and converts arbitrary text into the appropriate call with over 86% reliability. Adding an ASR model in front creates voice-to-function calling. Brick says synthetic data is central to this workflow. The team has found that roughly 10,000 to 10 million generated samples can be enough for high reliability, depending on the model and task. The Mobile Actions dataset is open sourced on Hugging Face.
Two fine-tuned tiny Gemma models enable offline voice dictation
Brick describes a production voice dictation app that performs all speech processing locally and does not require a subscription. One tiny Gemma model handles ASR, and another handles text policing. The app cleans up ums and ahs, supports personal biasing toward relevant words and names, and produces more polished text. Both models are fine-tuned and have parameter counts in the low hundreds of millions. This design moves a feature that might otherwise require a server and subscription onto the device. Brick also mentions Chrome developer-preview features for summarization and proofreading, where tiny models allow the features to reach more Chrome users.
"If we want intelligence to get into lots and lots and lots of devices and not just really expensive robots, we are going to need tiny models."00:01
Who should watch
You are choosing between cloud inference and local inference for a mobile, browser, IoT, or robotics feature.
Your device cannot spare several gigabytes of RAM, or the model is only one small part of a larger application.
You want a concrete fine-tuning workflow for speech, function calling, dictation, or another narrow task.
