Training should remain in one fast, tightly connected cluster, while rollout serving can spread across regions and providers.
2
Adam updates are dense in FP32, but most are too small to cross the rounding boundary of the lower-precision weights used by rollout engines.
3
Lossless patches can reduce a roughly 500 GB checkpoint transfer to roughly 500 MB and let Stitch coordinate version-aware rollout engines globally.
Summary
Nan Jiang argues that reinforcement learning post-training puts two different workloads into the same cluster even though they have different communication needs. Training needs fast all-reduce and model-parallel communication, while rollout workers mainly exchange policy versions, trajectories, and metadata. The rollout fleet can therefore move across regions and providers if it can reconstruct each served model version exactly. His proposed synchronization unit is a lossless patch over the rollout engine's visible weights. In BF16, a typical Adam update is far smaller than the rounding boundary, so the master weights change while the served values remain unchanged. The gradients and FP32 updates are dense, but the visible rollout changes are sparse. Lower-precision serving makes this effect stronger. Jiang describes a bulletin-board protocol and a sidecar that applies missing version transitions, with Modal's Stitch as an implementation. The approach could make scattered inference GPUs function as one elastic rollout fleet, though its behavior with Muon and fully asynchronous RL remains open.
Training and rollout serving need different hardware shapes
Nan Jiang separates the reinforcement learning loop into a tightly coupled trainer and a fleet of rollout serving jobs. The trainer performs backpropagation, all-reduce, and model-parallel communication, so it needs one region with a fast RDMA fabric. Rollout workers generate trajectories, call environments or tools, and send data back. There is no global all-reduce across rollout jobs. Jiang calls the trainer's shape a "cathedral", with many GPUs connected in one fast island, while useful inference capacity is spread across providers, regions, prices, and availability. Requiring both workloads to stay together means rollout capacity inherits the trainer cluster's hardest capacity constraint.
The rollout serving island can move out of the trainer cluster
The unit Jiang wants to move is a rollout serving island, which can be one engine or a local group of endpoints serving one policy version. A large model can still use local parallelism and local serving constraints inside an island. Across islands, the dependency is much lighter: the trainer sends a policy version, and rollout workers return trajectories and metadata. Training remains inside its fast cluster. Rollout islands can instead be placed in different regions or with different providers, as long as they can serve the requested version and report the version used for each request.
A full checkpoint is the wrong synchronization unit
A frontier-scale checkpoint can be around 500 GB, and moving it over a commodity network can take minutes to hours. That conflicts with the goal of getting weight updates to rollout workers within seconds, especially in asynchronous training. Jiang's proposal is to compare the rollout-visible weights from consecutive versions and transmit only the changed values. The rollout engine still has to reconstruct exactly the same served version that a full checkpoint would produce. The proposed object is a lossless patch containing changed positions, replacement bits, and metadata, rather than an additive floating-point delta that could accumulate drift.
The mechanism combines the rounding floor of the served format with the size of an Adam update. An optimizer may keep high-precision master weights, while the rollout engine serves a BF16 view. For a weight around 1, the nearest BF16 rounding boundary is roughly theta over 256, or about 0.0039. Jiang gives a typical Adam step at RL post-training learning rates of around three millionths. That update is more than a thousand times smaller than the boundary, so the master value moves without changing the served BF16 value. He calls this "Adam absorption". The effect is stronger in lower-precision serving because the rounding boundaries are farther apart.
The sparse served patch does not come from sparse gradients
Jiang directly rejects gradient sparsity as the explanation. The gradients are dense, with about 99% of parameters receiving nonzero gradients, and the FP32 master update is dense as well. The updates are simply too small to change the rollout engine's lower-precision view. Measurements that cast consecutive weights to BF16 and compare them bitwise found that around 99% remain identical per step across model families. The unchanged portion remains small even when rollout workers lag behind and need to apply several transitions. The patch is lossless, so changed positions and replacement values reconstruct the exact target version.
Lower-precision serving makes visible changes rarer
The same reasoning applies beyond BF16 to formats such as FP8 and NVFP4. For fixed floating-point formats, the visibility threshold depends on the format's mantissa. Lower precision generally places the threshold farther from the current value, so fewer updates cross it. Group-scaled formats such as INT4 need a slightly different analysis because each value shares a group scale, while NVFP4 uses hierarchical scales and different encoding details. Jiang gives an internal run serving GLM 4.7 Air in FP8: 0.15% of weights changed on the first step, when the learning rate was high, and the curve later settled near 0.05% per step.
A version-aware sidecar can coordinate scattered engines
The proposed system publishes immutable rollout versions to a shared bulletin board after each optimizer step. Each rollout engine knows which version it serves and which versions it can accept. Requests include the version being sent and the version the caller will accept, while responses include the served version, tokens, log probabilities, router replay information, and other metadata. A sidecar makes an ordinary rollout engine version-aware. If the engine is already current, the sidecar proxies the request. If it is behind but can catch up, the sidecar applies the missing transitions. If it cannot reach an acceptable version, it returns not ready.
Stitch turns the protocol into a global rollout fleet
Modal's implementation of this design is called Stitch. On the trainer side, Stitch publishes the rollout weight version. On the rollout side, agents pull the latest weights and synchronize them across regions and providers. The framework is intended to work across trainers, serving engines, and transport choices. Jiang says the transfer can fall from roughly 500 GB for a full checkpoint to roughly 500 MB for the patch, which makes synchronization possible in seconds. The rollout engines can then autoscale globally, allowing otherwise scattered inference capacity to act as one fleet. Jiang leaves open whether the same approach works for Muon, fully asynchronous RL, pre-training, mid-training, or SFT.
"The problem here is not whether rollout can leave the cluster, the problem is the full checkpoint is a wrong unit of synchronization."05:16
Who should watch
You are building RL post-training systems and your rollout GPUs are often in different regions or on different providers.
Your trainer cluster is available, but adding more tightly connected training nodes is difficult or expensive.
You need rollout workers to apply model updates exactly while keeping synchronization latency low.
