The spread of GPUs, CPUs, accelerators, and model-serving systems has made the AI software stack difficult to program and maintain.
2
Modular built Mojo, MAX, and Modular Cloud as one programmable stack for portable inference across different devices and deployment environments.
3
Abdul Dakkak reports faster results than competing systems on language, image, and multimodal workloads, while inviting users to test the open-source kernels and install Modular with pip.
Summary
Abdul Dakkak argues that the shortage of compute has accelerated a Cambrian explosion of AI hardware, with Nvidia, AMD, Intel, Arm, Apple, and other systems targeting different use cases. Existing software has become fragmented across vLLM, SGLang, TensorRT-LLM, llama.cpp, and hardware-specific layers. Modular rebuilt the stack around Mojo, a Python-like systems language for CPUs, GPUs, and accelerators; MAX, an inference framework for multiple model types and devices; and Modular Cloud, which exposes deployment and performance controls. Dakkak describes support for thousands of model architectures, multi-GPU and multi-node execution, and mixed hardware for tasks such as prefill and decode. He also presents benchmark claims for Kimi K2.5, FLUX.2, AMD MI355, and Gemma 4, including sub-second image generation. The talk ends with open-source kernels and model definitions, a planned open-source Mojo release, and instructions to try the software with pip.
The shortage of compute has created a wide range of competing hardware
Abdul Dakkak begins with the continuing shortage of compute, saying that much of it is booked for years. At the same time, hardware options have multiplied. He describes this as a "Cambrian explosion," with Nvidia GPUs across several generations alongside systems from Intel, Arm, Apple, and others. Each platform has its own niche. The resulting question is how developers can program all of these systems without rebuilding their software for every device. Dakkak presents hardware variety as the starting point for Modular's software design rather than as a temporary problem.
The current inference stack pushes hardware differences upward into applications
Dakkak says the software stack is difficult because hardware choices affect every layer above them. A diagram of vLLM supporting Nvidia and AMD becomes more complicated as more hardware is added. Teams also deploy with SGLang, TensorRT-LLM, and llama.cpp, each handling parts of the problem in different ways. Modular's response is to rebuild the stack from first principles around portability, performance, and programmability. The proposed stack unifies hardware behind a common layer, while still allowing developers to control the system and deploy it on premises or in the cloud.
Mojo combines Python-like programming with low-level hardware access
Mojo is Modular's heterogeneous systems programming language for AI and general-purpose work. Dakkak describes it as Pythonic, saying that people who know Python already understand much of its model. It targets CPUs, GPUs, and accelerators, and Modular positions it against C++, Rust, CUDA C++, OpenCL, and Triton. Dakkak says CUDA and related tools offer high performance with less usability and portability, while OpenCL offers portability with weaker performance and usability. Mojo aims to let developers write portable, usable kernels while retaining low-level access for maximum performance.
Modular has open-sourced kernels and claims strong results across hardware
Dakkak says Modular has open-sourced practically all of its kernels so people can test and benchmark them on their own machines. He claims the matrix multiplication kernels are faster than vendor libraries such as cuBLAS and hipBLAS, and that Modular's attention kernel is faster than FlashAttention 4. He also says Mojo 1.0 is being stabilized and that Mojo itself will be open sourced in the following months. These claims are presented as practical tests developers can verify, rather than as results that require access to a private system.
MAX replaces layers added around PyTorch for modern inference
MAX is Modular's inference framework, built around assumptions that Dakkak says PyTorch did not make when it was created about ten years earlier. Modern generative AI inference needs features such as KV cache handling and paged attention, which have led users toward systems such as vLLM and SGLang. MAX combines graph compilation, eager execution, Mojo-aware fusion, device portability, and model serving in one system. Python remains the language for developing models, and eager execution remains available for debugging. Dakkak describes this as a single stack for developing, serving, and optimizing models.
MAX supports many model architectures and mixed distributed hardware
Dakkak says MAX supports thousands of model architectures, including Gemma 4, MiniMax 3, and GLM, with an interface that follows current conventions. It also provides OpenAI-compatible endpoints. The APIs cover multi-GPU and multi-node execution, so distributed models can be programmed within the same framework. Dakkak specifically describes heterogeneous execution, where prefill can run on one type of hardware and decode on another. This is intended to let a deployment combine different devices instead of requiring every stage of inference to use the same GPU or accelerator.
Modular Cloud exposes deployment choices instead of hiding them
Modular Cloud sits above the language and inference layers for distributed deployments. Dakkak calls it a "glass box" because users can adjust variables across MAX, Mojo, and the endpoint configuration. It supports endpoint deployment and bring-your-own-cloud deployments. Modular Cloud Lab lets users experiment with parameters and select a point in the performance and cost tradeoff for a model or workload. The basic workflow resembles other cloud providers: choose a model, obtain an API key, and send HTTP requests. Users can also deploy dedicated endpoints and custom models, including models with custom speculative decoders.
The reported benchmarks cover language, image, and AMD workloads
Dakkak presents comparisons with what he calls the best combinations of vLLM or SGLang. For Kimi K2.5, he reports 5.5 times faster time to first token than vLLM and 1.5 times higher throughput per GPU on B200. For FLUX.2, he reports a result about seven times faster than diffusers compiled with torch.compile. On AMD MI355 systems, he reports four times the speed of the equivalent system and says AMD hardware is usually 30 percent cheaper than the Nvidia equivalent, so speed alone is not the full comparison. He also cites independent verification from Artificial Analysis.
The claimed speed makes interactive image generation possible
Dakkak connects the benchmark results to applications that slow inference can prevent. In the demonstrated image-generation workflow, prompts produce images in under a second in many cases, with examples around 0.3 seconds. He describes a booth demo that reads a video feed, accepts a prompt, and renders the result. He also reports that MAX is faster than other systems on Gemma 4, a model introduced about a month or a month and a half earlier. He says the reported results are early and that Modular continues to add optimizations.