Local inference is mainly constrained by memory capacity, memory bandwidth, and energy use during decode.
2
A full-stack approach could produce roughly 100x better price-performance for local frontier models through improvements to models, kernels, hardware, and orchestration.
3
Heterogeneous clusters can run prefill on compute-heavy hardware and decode on memory-rich hardware, as the demo showed with an RTX Spark and MacBook.
Summary
Alex Cheema explains EXO Labs' effort to run frontier models on local hardware. He argues that inference has received less optimization than training, leaving substantial gains in kernels, model design, orchestration, networking, and hardware. Decode is usually memory-bound, so local systems need enough memory to hold the model, high memory bandwidth, and low energy cost per byte moved. Prefill is compute-bound and can be separated from decode. EXO's approach combines devices with different strengths, such as Mac hardware for memory capacity and Nvidia hardware for compute. The workshop demonstrates a trillion-parameter GLM 5.1 model across four Mac Studios connected by Thunderbolt 5, using RDMA to reduce communication latency. A second setup sends prefill to an RTX Spark and decode to a MacBook, reducing end-to-end inference time for a large prompt from about seven seconds to 4.8 seconds. Cheema is candid that local hardware remains awkward and that benchmark quality needs more transparency.
EXO Labs is working on the full stack needed for local frontier inference
Alex Cheema describes EXO Labs as a lab focused on running frontier AI on local hardware. Its work covers models, software, and hardware, with the aim of lowering the cost of local inference. He separates this problem from training. If running a frontier model requires a million dollars of hardware, access remains limited even when the model itself is available. Cheema argues that much research has been shaped by the hardware used for training, especially stacked Nvidia GPUs, while inference may benefit from different designs. He sees substantial unexamined performance in the inference stack.
Kernel fusion can recover performance that looks lost to hardware limits
While running Qwen 3.5 on Apple silicon, EXO found that practical speed was about 50% below the theoretical estimate. The cause included unnecessary kernel launches. Each separate kernel added overhead, which matters when a model could theoretically generate around 150 tokens per second and each token takes less than 10 milliseconds. EXO fused operations that had been launched separately and increased inference performance by 30%. Cheema uses this as evidence that apparently optimized local inference software can still leave substantial performance unused. He says similar gains may exist in orchestration, communication, and other parts of the stack.
Local inference is usually a memory problem during decode
Cheema contrasts training with local inference. Training is generally compute-bound, so the cost and energy of floating-point operations matter. Inference, especially at low batch sizes, is mostly memory-bound because a local user cannot combine many users' requests into a large batch. Decode generates tokens one by one and repeatedly reads model weights and KV caches. The model must fit in memory, or loading it from disk becomes very slow. Memory bandwidth determines how quickly the data can be moved, while energy per byte determines how much power the inference consumes. These constraints make large unified-memory systems attractive even when their compute is limited.
Prefill and decode need different hardware characteristics
The prefill phase processes the prompt and creates the KV cache. Cheema describes it as compute-bound. Decode then generates tokens one by one and is memory-bound. He argues that prefill often matters less in ordinary local workloads because a good harness can reuse much of a stable system prompt and its tools. Large benchmark prompts can make prefill look more important than it is in typical use. This distinction allows a system to place each phase on hardware suited to it. Compute-dense Nvidia hardware can handle prefill, while a device with a larger memory pool can handle decode.
Intelligence per joule combines model quality with local operating cost
Cheema discusses a metric he calls 'intelligence per joule'. It measures how well a model performs on a task divided by the energy it uses. He says the term 'intelligence per watt' is less precise because the relevant question is the total energy needed for the task, rather than how quickly the energy is used. He cites work from a Stanford group associated with Hazy Research and says the metric had improved by about 5x over the previous two years. In his account, hardware improvements contributed about 5x and model improvements another 3x, with the gains compounding across the stack.
A local frontier model still needs expensive hardware today
Cheema uses GLM 5.1 to show the current cost of local frontier inference. He describes it as a newly released, trillion-parameter open model with FP16 weights requiring roughly 1.5 terabytes of memory. Running it across four 512 GB Mac Studios costs about $40,000 in hardware and reaches roughly 20 tokens per second. He says this is slower than what many users now expect from cloud services. EXO's thesis is that changes to the harness, models, kernels, and hardware could together create about 100x better price-performance. He estimates that a roughly $5,000 machine with near-frontier performance could be available within two years, while saying he would not recommend buying such a setup today just for ordinary use.
Most consumer workloads may stop benefiting from ever-larger models
Cheema expects local and cloud workloads to separate. Many consumer tasks follow an S-curve in which additional model intelligence eventually brings little extra utility. He uses transcription, summarization, email summaries, and to-do lists as examples. These tasks may reach a level where a smaller local model is good enough, while highly difficult scientific or medical problems could still require enormous data-center compute. He also thinks model specialization could become more practical if use cases stabilize. Today, rapidly changing frontier models make specialized chips risky. Later, several task-specific models and chips could be more efficient than one large model that handles everything.
EXO uses heterogeneous clusters and faster links to make large models practical
EXO runs as an app on each device and automatically discovers other devices in a mesh network. Its software chooses how to distribute a model based on the available hardware. Four Mac Studios in the demo are connected with Thunderbolt 5. Cheema says EXO's RDMA work reduced node-to-node latency from about 300 microseconds to single-digit microseconds. This matters for tensor parallelism, where a model with 60 layers may require two synchronizations per layer, or 120 synchronizations for each generated token. The lower latency reduced communication from roughly 30 milliseconds to less than one millisecond in his example, allowing the cluster to scale more effectively.
Benchmarking needs model quality and configuration details alongside speed
Cheema criticizes local inference comparisons that omit quantization, pruning, or other changes. A heavily quantized one-bit model may run a large model on a laptop, but he argues that a smaller unquantized model could be more useful. EXO plans to publish open benchmarks across different hardware, models, quantizations, and pruning choices. The planned site would combine raw measures such as tokens per second and prefill time with model quality and energy use. Users could select a hardware budget and compare setups on a Pareto frontier, such as a larger model at 20 tokens per second versus a smaller model at 100 tokens per second. He says the goal is to make progress and tradeoffs easier to inspect.
