Muna compiles plain Python inference functions into self-contained native binaries that can run across cloud, desktop, mobile, and web environments.
2
The compiler traces Python code into an intermediate representation, propagates types through that representation, and generates C++ or Rust code for native compilation.
3
LLMs generate the large set of elementary native operations, while tracing, compilation, verification, and testing constrain what generated code can do.
Summary
Yusuf Olokoba describes Muna's approach to deploying open-source AI models without requiring developers to rebuild infrastructure for every model or runtime. Developers write an ordinary Python inference function, such as one using Google's 270-million-parameter Embedding Gemma model. Muna traces that function into an intermediate representation, propagates type information through its operations, and generates C++ or Rust code. The result is a compiled library containing the model and its dependencies. It can be loaded from another language with FFI and exposed through an OpenAI-style client, including an embeddings.create method. Olokoba explains that early experiments used PyTorch FX and then LLMs to produce traces. PyTorch was too focused on PyTorch data types, while LLM tracing was accurate in testing but too slow. The current system analyzes Python's abstract syntax tree and uses internal heuristics. LLMs are used later to generate native implementations for the many elementary operations needed by the compiler.
AI deployment often creates infrastructure work around every new model
Olokoba begins with the routine problems facing AI engineers: an OpenAI client, several Hugging Face tabs, multiple playground repositories, and agent workflows built from HTTP calls. Trying a new open-source model usually means writing a Dockerfile, starting a container, and deploying it on owned or rented infrastructure. Adding that model to an agent creates another tool and context-management problem. Developers want a client that can point to models running locally or remotely, across runtimes such as Llama C++ or TensorRT, with few code changes.
Muna wants Python inference code to become a portable native binary
Muna's proposed workflow starts with a plain Python function and turns it into a small, self-contained binary. Olokoba says the binary should run in the cloud, on Apple silicon, or on other target devices. The compiled artifact contains the model and the dependencies it needs. The model can then be consumed through the familiar OpenAI client experience. This approach also supports the hybrid inference model he expects to see, where smaller models run near users or on their devices while larger cloud models handle more demanding reasoning.
The example uses Embedding Gemma to show the full compilation path
The worked example is a Python function that accepts a list of sentences and runs Google's Embedding Gemma 270-million-parameter model. It adds prefixes, tokenizes the text, runs the model, and returns embedding vectors or an embedding matrix. Olokoba connects this kind of model to text search, retrieval-augmented generation, and document retrieval. Because the model is small enough for consumer hardware as well as cloud GPUs, it provides an example of the broader goal: generate equivalent lower-level code, compile the model and its dependencies into a binary, and call it through an OpenAI-style embeddings.create interface.
Tracing turns Python execution into a graph the compiler can process
The first compiler stage is tracing, which produces a graph describing what the Python function does. Muna initially used PyTorch 2, Torch Compile, and Torch FX. Torch FX runs the function with fake inputs that do not allocate memory, then records a graph of the operations. That approach did not cover arbitrary Python code well because its tracer was focused on PyTorch. Supporting NumPy, OpenCV, and other data types would have required extending PyTorch. Fake tensors were easy to create, while fake images, dictionaries, and other real-world values were much harder.
Muna tested LLMs as trace generators because models could return structured outputs that matched a requested schema. Olokoba says this worked with almost 100 percent accuracy in their own testing, but it took too much time. The production direction became a conventional compiler approach: analyze Python's abstract syntax tree and apply internal heuristics to construct an intermediate representation. In the example representation, input nodes describe the list of strings, one node calls the tokenizer, another calls the model, and the outputs are returned to the caller.
Type propagation makes dynamic Python translatable to C++ and Rust
Python variables can change type, while C++ and Rust require declared types that remain fixed. Muna addresses this by propagating type information through the intermediate representation. In the example, the function signature says that the input is a list of strings, and the prefix map contains strings. The compiler therefore knows that concatenating a prefix with an input string produces another string. It can generate a native function that accepts two strings and performs Python's addition operation, then use that result type when processing later operations.
LLMs generate elementary native operations instead of entire arbitrary programs
The compiler still needs native implementations for many Python operations, from addition and subtraction to exponentiation and operations from libraries such as NumPy or PyTorch. Olokoba says the apparent scale of Python code comes largely from combining a smaller set of elementary operations in many ways. Muna can therefore cover the base operations and compose them in C++ or Rust. LLMs generate the native code for those operations, which avoids manually rewriting every implementation. Once types have been propagated through the graph, the compiler has enough information to generate C++ source that should compile.
FFI and an OpenAI-compatible wrapper make the compiled library usable
After native compilation, the output is a dynamic library or shared object that can be loaded into a process. Olokoba demonstrates using JavaScript on Node.js and FFI to declare the library, its function name, and its native signature. The JavaScript application can then load the compiled library and receive the embedding matrix. Muna wraps this in a client class with an embeddings class and a create function that mirrors the official OpenAI path. The model name resolves to the compiled binary's location, and the wrapper reshapes the result to match the official client's output.
"With this entire system in place, we have just recreated the official OpenAI client, but given it access to any open-source model that we can get into a Python function."17:15
Who should watch
You are deploying open-source models and want one client interface across local and remote runtimes.
Your Python inference code needs to run on devices or platforms where Docker and a full Python environment are a poor fit.
You are building compiler or code-generation infrastructure and want a concrete example of where LLM-generated native code can fit inside a constrained pipeline.
