# Luminal: Search-Based Deep Learning Compilers

Joe Fioti, Luminal | AI Engineer World's Fair 2025 | 24:35

Source: https://www.youtube.com/watch?v=0uj9lMI-sIo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/luminal-search-based-deep-learning-compilers
Published: 2025-06-03
Tags: gpus, inference, open-source, search

## TL;DR
- Luminal reduces its core representation to 12 simple operations, then uses compilers to turn primitive graphs into efficient machine-learning programs.
- Its compiler searches through logically equivalent GPU kernels, profiles their runtimes, and can discover optimizations such as kernel fusion and flash attention.
- The same graph representation supports inference, training through an external autograd library, and planned work across more hardware, distributed execution, reinforcement learning, and cloud inference.

## Summary
Joe Fioti presents Luminal as a small deep learning library built from simple linear-algebra operations. Instead of implementing a large collection of specialized operators for every device and data type, Luminal represents models as graphs built from 12 primitives. Compilers then transform those graphs into faster programs. Its main compiler uses rewrite rules to create a search space of equivalent GPU kernels, measures candidates, and selects fast implementations. This can discover optimizations such as kernel fusion and flash attention. Luminal also applies deterministic passes for buffer reuse and dispatching kernels in batches. Although it began as an inference library, an external autograd engine can derive backward graphs, allowing the same compiler pipeline to support training. Fioti describes current CPU, CUDA, and Metal support, along with plans for more hardware, distributed execution, reinforcement learning, and a cloud service that turns exported graphs into serverless inference endpoints.

## Key ideas
### Deep learning models can be built from a small set of operations
[00:24](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=24s)
Fioti argues that deep learning is fundamentally linear algebra over scalars, vectors, matrices, and tensors. Luminal therefore uses 12 primitive operations: unary operations such as exponential, logarithm, power-of-two, reciprocal, and square root; binary operations including addition, multiplication, modulo, and less-than; and reductions such as sum and maximum. More familiar operations can be composed from these primitives. Division uses multiplication and reciprocal. Matrix multiplication uses broadcasted multiplication followed by reduction. Convolution uses shape manipulation and a matrix-multiplication pattern. Fioti says this basis can represent language models, vision-language models, CNNs, RNNs, and diffusion models.

### Static model graphs allow Luminal to avoid much of the usual framework complexity
[04:35](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=275s)
Fioti says existing libraries were designed around dynamism because researchers once needed to experiment with models such as RNNs and LSTMs. He argues that most modern deep learning models are more bounded. In a transformer, the changing parts are mainly the KV-cache length and sequence length. Luminal represents the rest as a directed acyclic graph of operations. A small graph can express a dense neural-network layer through a weight multiplication and reduction, while larger graphs specify full models. The result is a library of under 5,000 lines of code that Fioti says should be understandable in an afternoon.

### Luminal compiles primitive graphs into high-performance programs
[06:43](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=403s)
The primitive graphs are intentionally slow and are not meant to run directly. Luminal sends them through compiler passes that produce faster graphs. This approach also keeps the software stack short. Instead of relying on several layers of handwritten kernels and libraries, Luminal directly emits CUDA code. Fioti contrasts this with stacks involving Hugging Face Transformers, PyTorch, xFormers, cuDNN, cuBLAS, and CUDA. Fewer layers make installation and debugging easier, since there are fewer dependencies between the model graph and the target runtime.

### Search lets the compiler handle optimizations that are too complex to encode by hand
[09:08](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=548s)
Fioti describes a limitation of conventional ML compilers: as generated kernels become more complex, compiler complexity can grow much faster than the kernels themselves. Luminal addresses this with a search-based approach inspired by AlphaGo. It converts model graphs into expressions in egglog, uses e-graphs to represent equivalent programs, and applies roughly 20 to 25 simple rewrite rules. Each rule preserves logical equivalence while adding another candidate to the search space. Luminal profiles candidate kernels and selects fast ones. When profiling every candidate becomes infeasible, it uses methods such as Monte Carlo search to reduce the space.

### The search can discover kernel fusion and flash attention
[15:02](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=902s)
Kernel fusion removes unnecessary trips to GPU memory. In Fioti's example, a naive graph loads a tensor, applies sign, writes it back, reads it again, applies a power-of-two operation, and writes the result. A fused kernel performs both operations after one load and before one final write. Fioti says data movement can account for about 99% of GPU energy and runtime in these workloads. He also describes Luminal's search finding flash attention from a naive multi-head attention graph. The compiler applies simple rewrite rules, builds and measures equivalent kernels, and selects the fastest result, which in this case is flash attention.

### Deterministic compiler passes reduce memory use and dispatch overhead
[18:17](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=1097s)
After search selects a kernel graph, Luminal applies optimizations that Fioti says are predictably helpful. Buffer reuse reduces memory use by assigning the same buffer to values whose lifetimes do not overlap. Since the complete workload graph is known, the compiler can determine when a buffer will never be used again. Luminal also dispatches all kernels ahead of time instead of waiting for the CPU to launch each GPU kernel after the previous one finishes. This removes repeated CPU-GPU launch round trips and lets the GPU process a prepared queue of kernels.

### An external autograd engine adds training without expanding Luminal's core
[20:35](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=1235s)
Luminal began as an inference library, but its graph representation also supports training. An external crate derives a backward graph from a forward graph and attaches the two. The existing search and inference compiler passes then run on the backward computation as well. Fioti presents this as a way to add training without making autograd part of the library's core. External contributors could write different autograd systems, gradient handling, or training setups against the same graph representation.

### Luminal's next targets include more hardware, reinforcement learning, and cloud inference
[21:35](https://www.youtube.com/watch?v=0uj9lMI-sIo&t=1295s)
Fioti says Luminal currently supports CPU, CUDA, and Metal. Planned targets include AMD, Tenstorrent, Groq, and TPUs, along with distributed inference and training using data, pipeline, and tensor parallelism. For reinforcement learning, the team wants to put both the model and environment into the Luminal graph so the GPU can run the forward pass and step the environment without repeated CPU-GPU transfers. The Luminal cloud lets users export a graph, upload it, and receive a serverless inference endpoint. Luminal handles optimization, batching, queuing, and machine provisioning, with payment based on graph execution.

## Notable quotes
- "The point isn't to run these primitive graphs of operations here. The point is to take these graphs and then run them through some functions to transform them into faster graphs." (07:05)
- "What we can do is we can write a whole bunch of simple rules to build this big search space and then let the search go through and find the fastest kernels." (12:58)
- "The fastest one in this case just happens to be flash attention." (17:17)
- "We basically get training for free." (20:54)

## Tools & references mentioned
- Luminal
- PyTorch
- TensorFlow
- Hugging Face Transformers
- xFormers
- cuDNN
- cuBLAS
- CUDA
- egglog
- AlphaGo
- FlashAttention
- Tri Dao
- Metal
- AMD
- Tenstorrent
- Groq

## Who should watch
- You are building an ML runtime or compiler and want to replace large collections of handwritten kernels with a smaller graph representation and automated search.
- Your inference stack has high memory traffic or CPU-GPU dispatch overhead, and you want to see how graph-wide compiler passes can address both.
- You are evaluating a lightweight alternative for training or inference across several accelerator types, including the possibility of exporting graphs to a hosted endpoint.

## Related talks

- [Compilers in the Age of LLMs](https://aietalks.com/talks/compilers-in-the-age-of-llms) (Yusuf Olokoba, Muna, 17:36)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [Llamafile: Bringing AI to the Masses with Fast CPU Inference](https://aietalks.com/talks/llamafile-bringing-ai-to-the-masses-with-fast-cpu-inference) (Stephen Hood & Justine Tunney, Mozilla, 17:25)
- [Can LLMs Write Fast Multi-GPU Kernels?](https://aietalks.com/talks/can-llms-write-fast-multi-gpu-kernels) (Simran Arora, Together AI, 30:00)
- [Why MLX](https://aietalks.com/talks/why-mlx) (Prince Canuma, Neywa Labs, 23:10)