# Compression at the Edge Chris Alexiuk, NVIDIA & Daniel Han, Unsloth & Asma Beevi, NVIDIA & Merve Noyan, Hugging Face & Parth Sareen, Ollama | AI Engineer World's Fair 2026 | 46:01 Source: https://www.youtube.com/watch?v=J4_jCrTxMkk Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/compression-at-the-edge Published: 2026-08-07 Tags: edge, evals, inference ## TL;DR - Quantization makes large models practical on personal computers, but every compression method trades away some quality, speed, or memory. - Layer sensitivity is highly uneven, so useful quantization keeps selected layers at higher precision instead of applying one setting everywhere. - Benchmark scores are only an initial check; comparing BF16 and quantized output logits with KL divergence can reveal how closely the compressed model follows the original. ## Summary The panel explains why quantization has become necessary as open models grow beyond the memory of ordinary computers. Daniel Han describes shrinking GLM 5.2 from 1.5 terabytes to 250 GB with dynamic, mixed-precision quantization, while preserving much of its capability by treating layers differently. Asma Beevi explains NVFP4, where groups of 16 values share an FP8 scale, and describes NVIDIA's use of sensitivity analysis and selective precision. The speakers also discuss QLoRA, local inference through Ollama, and the practical limits of benchmark scores. New architectures make compression harder, especially when linear attention layers appear to work on ordinary tests but fail on long contexts. The panel expects compression to move toward phones, KV-cache compression, sparsity, and more varied model architectures. Han prefers KL divergence over output logits as a direct comparison between BF16 and quantized models. ## Key ideas ### Compression makes large models usable on ordinary hardware [00:57](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=57s) Parth Sareen describes quantization as one of the reasons Ollama could run larger models on relatively small machines. Merve Noyan says compression democratizes models by bringing them to edge devices and personal computers. Asma Beevi frames it as getting more intelligence at the same cost, pointing to the move from FP32 training toward FP4. The panel also stresses that compression has a cost. That cost can appear as lower quality or higher latency. The practical goal is to shrink a model without making it useless, so people can run capable systems locally instead of relying only on hosted infrastructure. ### Mixed precision can shrink GLM 5.2 without destroying its capability [03:05](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=185s) Asma Beevi gives GLM 5.2 as an example of a model that is about 1.5 terabytes before compression and about 250 GB afterward. Daniel Han says dynamic quantization can leave selected layers at higher precision while putting most layers at one or two bits. Some especially important layers remain at 16 bits. He says this approach recovered 76% of all accuracy in their example, rather than making an 86% smaller model 86% worse. The result depends on choosing which parts to compress, rather than applying the same precision to every layer. ### Layer importance is uneven because many weights remain unsaturated [12:32](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=752s) Han says the first and last layers of a language model can be very important, while many middle layers are less important. His explanation is tied to training scale. A model with one trillion parameters may be trained on 30 trillion tokens, yet its weights are still not fully saturated. Some weights remain close to zero and can be set to zero. Finding safe choices requires testing layers individually and then testing combinations. The search becomes a combinatorial optimization problem because the relevant choice may involve several layers, tensors, or individual values at once. ### A single super weight can have an outsized effect on model quality [14:20](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=860s) Han cites the super weights paper, which found that quantizing one particular number in the entire model can make the model 20% worse. This means a quantizer has to identify values that cannot tolerate reduced precision. The problem is more detailed than deciding which layer to preserve. Within a layer, particular tensors or individual numbers may have unusual importance. Han uses this finding to explain why random rounding or randomly selecting model parts for compression can fail completely. Successful quantization depends on understanding the model's internal sensitivity. ### NVFP4 uses shared scaling to make four-bit values useful [16:37](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=997s) Beevi explains that NVFP4 is a four-bit floating-point format with microlayer scaling. A group of 16 elements shares one additional FP8 scaling value. That scale lets the four-bit values represent their local range more effectively than simply storing smaller numbers. NVIDIA uses NVFP4 for weight compression and four-bit matrix multiplication. Beevi says the format preserves more accuracy than other four-bit formats in their experiments. The panel's explanation makes clear that the format includes both the four-bit values and the shared scale, so the representation is more structured than a simple reduction in bit count. ### Real use can expose failures that standard benchmarks miss [17:55](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1075s) Parth Sareen says businesses can benefit from compression even when they have access to larger infrastructure. A compressed model can run on individual employee machines, increase concurrency, or support in-house deployment. He also says a model can look acceptable in benchmarks and still feel wrong inside an actual harness such as a coding workflow. Merve Noyan adds that many benchmarks cover verifiable tasks, while model behavior in practical use is harder to capture. The speakers treat benchmark scores as a direction check. They want people to run the quantized model in the environment where it will actually be used. ### A larger compressed model can beat a smaller full-precision model [21:49](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1309s) Han describes experiments comparing a smaller model in BF16 with a model several times larger compressed to four bits, while keeping the two models at a similar disk size. He says the larger quantized model performed better in intelligence tests. The trade-off depends on the use case. A compressed large model may be slower if the available GPU is limited, while a small model can produce many more tokens per second. Han also suggests routing tasks between models, using a larger model for planning and a smaller one for execution. Size, speed, throughput, and available hardware all affect the choice. ### New attention architectures create new quantization failures [32:46](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=1966s) The panel says model architectures have moved beyond a mostly shared Transformer design. Current models may combine sliding-window attention, global attention, linear attention, sparse attention, and other variations. Han says older quantization heuristics do not transfer reliably. In one example, linear attention layers looked fine under ordinary checks, but quantizing them caused the model to produce gibberish on long-context benchmarks and in production. Some layers can be reduced to very low precision, while others must remain at higher precision. Ollama also has to test both model implementations and default quantization choices so users receive a model that works in real harnesses. ### KL divergence offers a direct comparison with the BF16 model [43:22](https://www.youtube.com/watch?v=J4_jCrTxMkk&t=2602s) When asked how to choose among many modified checkpoints, Han says comprehensive evaluation is not yet available. He prefers KL divergence, which measures the distance between the output logits of the original BF16 model and the quantized model. The process feeds calibration data through both versions and compares their output logits. The quantizer's goal is to reduce that distance toward zero while also reducing model size. Han presents this as a cleaner signal than accuracy alone because accuracy requires sampling choices, repeated trials, and averaging. It does not replace practical testing, but it gives quantization work a direct numerical target. ## Notable quotes - "There is absolutely zero free lunch." (01:41) - "If you quantize the correct layers you will not make the model literally useless." (03:47) - "If you quantize one number just one of the entire model your model becomes 20% dumber." (14:20) - "If you quantize the linear attention layers, it looks like it's doing good but then when you do long context benchmarks it becomes gibberish." (36:23) - "Your goal is to make the distance zero and the size smaller." (45:13) ## Tools & references mentioned - Ollama - Unsloth - Hugging Face - NVIDIA Model Optimizer - NVFP4 - BF16 - FP8 - QLoRA - TRL - bitsandbytes - Llama.cpp - GLM 5.2 - DeepSeek R1 - Qwen 3.6 - Gemma 4 - NeMoTron - Super Weights paper ## Who should watch - You want to run a large open model on a laptop, phone, Mac, gaming computer, or other limited hardware. - Your team needs to choose between a smaller full-precision model and a larger quantized model for local or private deployment. - You are building quantization pipelines and need to test layer sensitivity, long-context behavior, and output-logit differences.