AI efficiency helps move systems from demos into products by reducing resource use and latency.
2
Small language models can deliver strong task performance while enabling deployment on clouds, laptops, mobile devices, and edge hardware.
3
Four-bit quantization can reduce memory and latency with little performance loss, but every quantized model still needs evaluation before deployment.
Summary
Shelby Heinecke presents efficiency as a practical requirement for deploying AI at scale. Cloud deployments pay for compute and storage, on-premises systems have limited GPUs, and mobile or edge devices impose tighter hardware constraints. She organizes the problem into five dimensions: efficient architecture selection, pre-training, fine-tuning, inference, and prompting. The talk focuses on two of them. Small language models use less memory and compute, run with lower latency, and can fit on devices beyond the cloud. Heinecke discusses Phi-3, MobileLLM, and an Octopus model fine-tuned for Android function-calling tasks. She then explains post-training quantization, which reduces model weights from formats such as 16 or 32 bits to fewer bits. Four-bit quantization can cut storage and improve latency with little measured performance change in the examples shown. Three-bit quantization caused a performance drop, so evaluation is required. She recommends llama.cpp, ONNX Runtime, and her team's Mobile AI Bench for testing quantized models, including device latency and hardware use.
Efficiency is the step between an impressive demo and a production system
Heinecke says new AI capabilities are often presented as demos or prototypes, while builders must handle scaling and production deployment. Efficiency helps close that gap. AI systems consume paid cloud resources such as GPUs, CPUs, and disk space. On-premises deployments face restricted GPU capacity, and mobile or edge deployments have even tighter hardware limits. She frames the problem around real deployment conditions, where resources are never infinite. For Salesforce, which deploys AI at Fortune 500 scale, the cost and capacity of the serving environment make efficiency a practical product concern.
AI efficiency has five dimensions that teams can consider
Heinecke groups efficiency into five directions. Architecture selection includes choosing small models, mixture-of-experts architectures, and efficient attention mechanisms. Pre-training can use mixed-precision training and scaling methods, although relatively few teams do their own pre-training. Fine-tuning can update only a subset of weights through methods such as LoRA and QLoRA instead of changing every weight. Inference can use post-training quantization and speculative decoding. Prompting also affects memory and latency, so she recommends concise prompts, template formatting, and prompt compression.
Small language models reduce resource use and expand deployment options
Heinecke defines the small language models she discusses as models with 13 billion parameters or fewer. Fewer weights mean lower RAM, GPU, CPU, and disk use, along with faster fine-tuning and lower latency during the forward pass. This makes small models suitable for cloud and on-premises servers as well as laptops, mobile devices, and edge hardware. She presents small models as a response to the practical limits of serving large models. Their value comes from fitting the target hardware while retaining enough task performance for a particular application.
Phi-3 shows that a smaller model can compete with a larger one
Heinecke discusses Phi-3, a 3.8-billion-parameter model. She says its technical report shows it outperforming a well-known 7-billion-parameter model on the comparison presented in her slide. The example is meant to challenge the assumption that useful performance requires very large models. A model with fewer parameters can reduce resource consumption while remaining competitive on the tasks it targets. Heinecke uses Phi-3 as an example of why teams should assess model size alongside task performance when choosing an architecture.
Fine-tuning can make very small models competitive on specific tasks
Heinecke then describes MobileLLM, which has 350 million parameters. She says that after fine-tuning, it is on par with a 7-billion-parameter model on the tasks evaluated. She also discusses an Octopus model fine-tuned from Gemma 2B for Android tasks. In the comparison shown, the fine-tuned 2-billion-parameter model outperformed GPT-4 and Llama 7B on those Android tasks. Her point is that model size alone does not determine usefulness. Fine-tuning for a defined task can make a very small model competitive, especially when deployment hardware is limited.
Quantization maps larger numerical values to lower-precision representations. Heinecke explains that language-model weights may originally use 32-bit or 16-bit floating-point formats. Quantization can reduce them to 8 bits, 4 bits, or fewer. A lower-bit representation takes less storage and consumes less memory, CPU, and GPU capacity. She illustrates this with Llama models at 7B, 13B, and 70B parameters, where four-bit versions occupy a fraction of the original disk space. The technique is especially useful when a model must fit within a constrained serving environment.
Four-bit quantization can improve latency without a large performance tradeoff
Heinecke points to a study of large multimodal models in which four-bit quantization reduced time to first token compared with the original 16-bit encoding. She also shows a vision-language task where performance at four bits was essentially unchanged from the higher-precision versions. In that example, the reduction in resource use and latency came without a meaningful loss on the measured task. She warns that quantization can go too far. The three-bit result in the same discussion showed a performance drop, so teams should measure the quantized model rather than assume that every lower-bit format is acceptable.
Quantized models need task, safety, and device evaluation before deployment
Heinecke recommends evaluating a quantized model before serving it. Her team's open-source Mobile AI Bench is intended to streamline evaluation across text tasks, trust and safety, and vision-language tasks. She specifically says teams should check that trust and safety performance does not degrade after quantization. For on-device deployment, the project also includes an iOS app that measures model latency and hardware use. The app can be used to inspect effects such as battery drain. She names llama.cpp and ONNX Runtime as practical frameworks, and notes that quantized model versions are often available through Hugging Face.
"The point here is that when you're deploying AI models you're deploying in these constrained environments, it's never we're never in a situation where we have infinite resources."08:44
Who should watch
You are choosing a model for a product and need lower serving cost, memory use, or latency.
Your team is considering mobile, laptop, on-premises, or edge deployment with limited hardware.
You have a quantized model and need a practical evaluation plan before shipping it.
