AI engineers who use or deploy models need to understand enough GPU hardware to make good performance decisions, even when they do not build GPUs or write infrastructure.
2
GPUs favor bandwidth and throughput over low latency, with their strongest performance coming from low-precision matrix-matrix multiplication on tensor cores.
3
Many inference techniques become much more affordable when they turn matrix-vector work into matrix-matrix work, such as generating multiple samples or multiple tokens at once.
Summary
Charles Frye argues that AI engineers need a practical understanding of GPUs for the same reason application developers need to understand database indexes. They do not need to build the hardware, but they need to know how to use it well. GPUs favor high bandwidth and throughput rather than low latency. Their strongest path is arithmetic work, especially low-precision matrix-matrix multiplication on tensor cores. This explains why prompt processing is more GPU-friendly than autoregressive decoding, and why small models can sometimes produce better economics when run many times with a verifier. Frye also connects GPU design to wider changes in software, including parallelism and concurrency, because latency has improved more slowly than bandwidth. His advice is concrete: look for ways to keep tensor cores busy, turn matrix-vector operations into matrix-matrix operations, and consider smaller models that fit on available hardware.
AI engineers need hardware knowledge even when they use managed model APIs
Frye compares AI engineering with application development around databases. Most developers do not build a database, and many do not run one, but they still need to write queries that use indexes effectively. He says language models are moving toward a similar point, where engineers will increasingly integrate models tightly or run them themselves. That requires enough GPU knowledge to make good use of the hardware without becoming a GPU designer. His one-sentence summary is, "use the tensor cores Luke." The practical target is understanding how to make model workloads fit the parts of the GPU that deliver the most throughput.
GPUs favor bandwidth and throughput over low latency
The primary design choice Frye asks engineers to understand is that GPUs embrace high bandwidth rather than low latency. They optimize for mathematical throughput, especially arithmetic bandwidth, instead of simply moving memory quickly. He narrows the preferred workload further: low-precision matrix-matrix multiplication, rather than matrix-vector multiplication. This changes how engineers should think about performance. A workload that waits for individual operations may leave much of the GPU unused, while a workload that keeps many calculations in flight can use the hardware more fully. The same broad tradeoff applies to TPUs, but differs from most hardware programmers encounter.
Software has shifted toward parallelism and concurrency because latency scaling slowed
Frye describes the end of the period when faster clock speeds improved programs without code changes. Clock rates stopped improving rapidly in the early 2000s, so software had to expose more parallel and concurrent work. Parallelism performs multiple operations in one clock cycle. Concurrency starts new work while earlier work is still running, which can require patterns such as async and await. GPUs push both ideas further than CPUs. Frye gives the example of an AMD EPYC CPU handling two threads per core, compared with more than 16,000 parallel threads on an NVIDIA H100. GPU context switching can happen every clock cycle through the warp scheduler.
Bandwidth improves faster than latency, so engineers should design around it
Frye calls the pattern "Patterson's law: Latency lags bandwidth." Across networks, memory, and disks, he says bandwidth has improved much faster than latency. For every 10x improvement in latency, the cited trend gives roughly a 100x improvement in bandwidth. Latency runs into physical limits, such as the speed at which network packets can travel. Bandwidth can grow by doing more things at the same time. This helps explain why GPUs, and the software used with them, emphasize keeping many operations in flight. Frye's advice is to bet on hardware that increases bandwidth when choosing how to structure AI workloads.
GPU arithmetic bandwidth matters more than memory bandwidth
A GPU may have high-bandwidth memory, but Frye says its strongest advantage is the amount of computation it can perform on data. This is captured by arithmetic intensity, or the amount of mathematical work done for each memory load. An algorithm with n-squared operations can work well when it needs only n memory loads. He applies this to language-model inference. During prompt processing, an 8-billion-parameter model may move about 8 gigabytes from memory and perform around 60 billion floating-point operations. Decoding has to move the parameters again for each step, so it makes much less favorable use of the GPU's arithmetic capacity.
Small models can gain an advantage when they generate many candidates
Frye describes a way to make decoding more compatible with GPU hardware: run a smaller model many times on the same prompt, then select an answer with a verifier. His example uses an 8-billion-parameter model to generate many outcomes and checks them with a Python test. He says this approach can match the quality of GPT-4o in the cited setup, using Llama 3.1 8B and 100 generations. The reason is that the model weights are loaded once while the GPU performs more computation on them. This gives smaller models an economic and hardware-efficiency advantage when a useful verifier exists.
Tensor cores reward workloads that turn matrix-vector work into matrix-matrix work
Frye says the tensor core is a large GPU component specialized for low-precision floating-point matrix multiplication. Language-model generation is naturally heavy on matrix-vector operations, which use the tensor core poorly. He cites microbenchmarks where a matrix paired with a mostly empty matrix containing one full column reaches about one-thousandth of the performance of a fuller matrix workload. Adding more samples or tokens can turn that underused operation into matrix-matrix multiplication. This is why multi-token prediction, multiple samples per query, and similar techniques can become approximately free relative to the work already being done. The hardware pushes engineers toward batch-like generation.
AI engineers should consider smaller models and GPU-aware deployment choices
Frye's practical recommendation is to ask whether a smaller model can fit on a GPU available locally, then scale out its generation to reach the quality users need. He says open models are now good enough for this approach to become useful again. The decision depends on more than model quality in isolation. Engineers should consider how much arithmetic the hardware can perform, whether the workload keeps tensor cores occupied, and whether a verifier can select among generated results. He points listeners to his GPU glossary, which explains terms across the CUDA hardware and software stack, and to Modal for running serverless GPU workloads.
"The specific thing we want to do is low precision matrix multiplication."16:09
Who should watch
You are building an AI application on model APIs and expect to self-host or integrate models more tightly.
Your inference bill or latency is hard to explain, and you need a way to connect workload shape with GPU utilization.
You are choosing between a larger model and a smaller model that can generate and verify many candidate answers.
