Code World Model learns from program execution traces so it can predict how code changes state, rather than learning only token sequences.
2
CWM is a 32-billion-parameter model trained on GitHub data, execution traces, long-context data, instruction-following examples, and agentic reasoning tasks.
3
The model can trace functions, fill in code around partially specified structures, and simulate program behavior before expensive execution.
Summary
Jacob Kahn presents Code World Model, or CWM, as a way to train models on computation rather than only on code syntax. The model represents execution as a sequence of program states, actions, local variables, and outputs. This gives an autoregressive model a way to predict what happens after each step. CWM uses GitHub repositories, pull requests, tests, CI runs, and generated execution traces, then trains a 32-billion-parameter transformer with long-context, instruction-following, and agentic post-training. Its tools are deliberately limited, with Bash as the main interface for changing files and running code. Kahn shows applications such as tracing variables, completing partially specified code, and simulating expensive programs or distributed systems. He also discusses asynchronous reinforcement learning, where samplers and trainers queue models and trajectories and can update a model during an active trajectory. He is careful that these simulations approximate execution rather than solving undecidable problems exactly.
A world model predicts future program states from actions
Kahn defines the goal of CWM as building models that can reason, plan, and make decisions. Code is a constrained environment because it has explicit rules. A world model takes past observations and actions and predicts future observations. In a code setting, the model can take an action such as executing the next line, observe the resulting program state, and use that transition for later reasoning. Kahn also rejects a strict split between world models and language models. A world model is a way to parameterize a problem, while a language model is one way to represent and use that parameterization.
Modeling code requires representing execution, not only syntax
A token-based code model sees syntax, predicts more tokens, and has no explicit representation of what happens when the program runs. CWM instead aims to predict program execution. Its traces can include frame separators, local variables, memory details, and the state after each line. Kahn gives the example of counting the letter 'r' in 'strawberry', where each execution step records what the program is doing. The same format could extend from one function to a whole repository, a distributed system, or a code-contest solution.
Execution traces turn program behavior into an autoregressive prediction task
CWM treats a program or data as a state, executing the next line as an action, and the result as the next state. A model can generate this transition sequence token by token. The trace can also be expressed in natural language, giving the model a systematic description of execution. In an agentic setting, the same pattern includes a problem, a proposed action, feedback from the environment, and another decision. CWM can use an imagined execution trace to estimate the result of an action before interacting with the real environment.
Repository data and execution traces connect training to real software tasks
The training data starts with a large amount of GitHub data, including repository-level events and pull requests. The team mutates pull requests, predicts changes, and runs tests or CI on repositories known to pass. Those runs provide execution traces that extend beyond small, isolated programs. This lets the model learn from software behavior at repository scale. Kahn describes CWM as a 32-billion-parameter dense transformer with a long context length. It is pretrained on a few trillion tokens, then receives domain-specific and long-context training, instruction-following and reasoning fine-tuning, and joint reinforcement learning with agentic tasks.
CWM learns software work through a Bash-centered environment
CWM interacts with tasks through an environment that returns tokens, rewards, log probabilities, compiler output, and other feedback. The model has fewer tools than some other systems, so it must learn to use the terminal well. In the setup called SWRL, an issue is paired with repository data and the model uses Bash commands to inspect and change files. It can eventually use an edit tool or create content, but the training environment is intended to resemble the terminal-based setting in which an engineer works. Failed agent traces can also be rejection-sampled and fed back into training.
Asynchronous reinforcement learning keeps training and sampling busy
CWM's post-training system has samplers that produce trajectories, an environment that executes terminal actions, and a trainer that scores trajectories and computes gradients. Since samplers can produce data faster than trainers consume it, the system queues both model checkpoints and trajectories. This allows the loop to remain relatively on policy without waiting for every operation to finish. Kahn says the model can even be replaced during an active trajectory. That makes the trajectory somewhat off policy, but the system's throughput and data volume allow the team to accept this risk while reducing idle time.
The model can trace code and complete structure-aware partial programs
CWM can trace a function line by line and report local variable values at points in the execution. Kahn uses this capability to describe a neural debugger. A programmer can write the structure of a program, leave a variable or section incomplete, and let the model infer the missing code from the surrounding control flow. The programmer does not need to describe every ambiguity in natural language. The model can use the written loop, condition, and assignments as a partial specification, then simulate the code to infer what the programmer is trying to make it do.
Simulation may approximate expensive computation without solving undecidable problems
Kahn connects CWM to questions such as whether a program halts. Exact solutions can require simulating an execution forever, so CWM cannot remove the underlying theoretical limit. It may still approximate program behavior by recognizing patterns in simulated execution. The same idea could help inspect a large distributed system or an expensive function without running the full system each time. The model's internal execution simulation gives it a way to reason about possible behavior, while its conclusions remain approximations rather than formal decisions about every program.
"We want to predict program execution because we believe it might lead to us better modeling things about code, writing code, analyzing code, and beyond."02:16
Who should watch
You are building code agents and want them to inspect repositories, run commands, and reason about execution states.
You are researching world models or program reasoning and want a concrete training setup based on execution traces.
You are designing reinforcement-learning infrastructure for coding agents and need an asynchronous sampler, environment, and trainer loop.
