Small and task-specific models can reduce security, latency, connectivity, energy, and inference costs for many jobs.
2
A production trace dataset and explicit success criteria let teams choose the smallest model that performs well enough.
3
Prompt variants, post-processing, and regression evals can close much of the gap between a local model and a frontier model.
Summary
RL Nabors argues that many calls sent to frontier models could run on smaller language models or task-specific models. Local inference can keep data on the device, work offline, reduce latency, and remove per-call API fees. She presents a practical process: prove the task is possible with a capable model, create a golden dataset from real inputs and expected outputs, define measurable criteria, then test models from small to large. In her Mima summarization example, Llama 3.2 was close to Claude Sonnet and became faster and cheaper after a few-shot prompt and simple post-processing. She also shows why aggregate scores need inspection, since an LLM judge can be overly strict. Regression evals then protect the feature when prompts or models change. The talk gives engineers a way to replace guesses about model size with measurements from their own application.
Remote frontier inference has costs beyond the API bill
Nabors groups the costs of cloud inference into security, latency, business spend, connectivity, and energy. Sending data to remote servers creates risks around exposure, interception, retention, breaches, and leaks. Research she cites puts four seconds at the limit of believable response delay for users in LLM chats, while many large-model calls take longer. Agentic workflows can also multiply calls and tokens, even as token prices fall. Remote models fail when the user is offline, in a secure environment, or during an outage. Local inference avoids network round trips and can keep information on the device.
Nabors distinguishes task-specific models and smaller language models from broad foundation models. For visual tasks she names MobileNet, YOLO, and MediaPipe. For audio she names Whisper and Wave2Vec2. For language tasks she mentions Gemma and Qwen. SLMs range from millions to billions of parameters, while larger models can reach trillions. A small model does not need the full breadth of knowledge learned by a frontier model when the job is summarizing a thread or classifying abusive behavior. Quantization to 8-bit or 4-bit can reduce disk and memory requirements. She gives one upper-bound estimate: a one-billion-parameter model fits in about 2 GB in FP16.
The right model is the smallest one that meets the measured bar
Nabors calls her selection rule 'prototype big, deploy small.' First, use the largest capable model to check whether the task is possible. Then collect representative inputs and outputs and define what success means. Finally, test models from small to large until one falls within an acceptable range. In her Mima example, Claude Sonnet produced thread summaries for a baseline. The evaluation set contained 14 threads and 28 examples because each thread had a short-summary outcome and a summary-with-references outcome. She measures JSON validity, reference validity, factual consistency, length compliance, and P50 and P95 latency.
A capability eval compares local candidates against a practical baseline
Phoenix is used to compare the smaller candidates with the large-model baseline. Nabors tested Qwen 2.5 Instruct at 1.5 billion parameters, Qwen 3 at 1.7 billion, Llama 3.2 at 3 billion, and Gemma 4 E2B at 5 billion. Qwen 2.5 had roughly one second of P50 latency, but its accuracy was low. Gemma 4 E2B took around eight seconds in her tests. Llama 3.2 reached about 90% accuracy and was much faster than Gemma 4, while Claude Sonnet remained the accuracy ceiling at around three seconds. Looking at the raw responses showed that Llama was often nearly indistinguishable from Claude.
Prompt changes can improve a local model, but their effects must be isolated
Nabors tested four prompt variants against the original Llama prompt. One reformatted the thread as numbered messages instead of JSON. One added a few examples. One added strict negative rules. One used chain of thought to identify key moments before writing. The numbered format made little difference, and explicit rules made the model worse. Chain of thought slightly improved length performance while adding 600 milliseconds. The few-shot prompt performed best. It improved length compliance and factual and reference agreement while adding only 200 milliseconds. Her method changes one variable at a time so the effect of each prompt can be measured.
Evaluation scores need inspection because judges can disagree with humans
The improved Llama result reached 91.7% structural validity and 92.9% factual consistency before post-processing, with P50 latency below 1.5 seconds. When Nabors inspected the factual-consistency cases, she found that Claude was acting as a very strict judge. It objected to small wording differences, such as describing someone as 'angsty' rather than 'cross.' That made the numerical gap look larger than the practical difference between the summaries. She recommends opening experiments and comparing raw responses, expected responses, and judge decisions instead of accepting aggregate scores without review.
Simple harness checks can remove avoidable model errors
Some requirements did not need a larger model. The harness could check whether the number of references exceeded the number of messages in a thread. It could also truncate a summary that exceeded the desired length. After adding this post-processing, Nabors reached 100% JSON validity and 100% structural validity. Factual consistency still had a small disagreement caused by the strict judge. P50 latency was around one second, and P95 latency was below three and a half seconds. The local implementation beat Claude on latency and saved about a dollar per day in inference costs for Mima.
Regression evals protect local features from future prompt and model changes
Nabors says the evaluation suite should continue running after deployment. A prompt update or model upgrade could otherwise make summaries expand into paragraphs or introduce false claims. She compares regression evals with CI/CD tests. Teams can use production inputs as a standing test set and rerun it whenever the model or prompt changes. She also points to models already available on user devices, including Gemini Nano through Chrome's Prompt API, since using an existing browser model avoids shipping another model. Her final process is to prove the task, define success, test candidates, select the smallest model that passes, and keep testing it.
"You're going to want to select the smallest model that gives acceptable responses for your use case."15:45
Who should watch
You are paying for frontier-model calls inside a product and need to find candidates for local replacement.
Your team has real production traces but no clear definition of model quality, latency, or acceptable failure rates.
You ship an agent or AI feature that could regress when someone changes a prompt or upgrades the model.
