Applied Compute trains enterprise models with reinforcement learning on private, task-specific benchmarks, so runs must be fast, cheap, and predictable.
2
Synchronous RL leaves GPUs idle while they wait for the slowest sampling request, especially when response lengths have a long tail.
3
Pipeline RL improves throughput by sampling and training at the same time, but stale samples increase the variance of importance ratios and can destabilize learning.
Summary
Rhythm Garg and Linden Li describe the systems work behind Applied Compute's reinforcement learning stack for enterprise tasks. Their customers need models trained on specialized data and delivered within days, so training runs must control both cost and completion-time variance. The talk starts with synchronous RL, where sampling and training alternate and the slowest sample determines each step. A workload using Qwen 30B shows how a small tail of long requests can leave GPUs underused. Pipeline RL overlaps sampling and training, which raises utilization, but introduces stale tokens because samples can span several policy updates. The speakers then build a first-principles simulator around GPU allocation, training and sampling throughput, response-length distributions, KV-cache limits, and a staleness threshold. The simulator searches for a viable split between training and sampling GPUs before an expensive run begins. In their example, an optimized asynchronous layout gives roughly a 60% speed-up over the synchronous baseline.
Applied Compute uses reinforcement learning to specialize models for enterprise work
Applied Compute builds systems for company-specific automations rather than general productivity tools. Once a system is specialized to how a company operates, the team deploys it with a data flywheel so performance improves through use. Garg describes reinforcement learning as the mechanism that moves models from out-of-distribution behavior toward the private, in-distribution tasks that matter to each customer. The team previously worked on reinforcement learning at OpenAI, where they saw it improve public benchmarks. Their current focus is applying the same basic mechanism to private benchmarks and business workflows.
RL reinforces successful reasoning traces through repeated attempts
Garg explains the training mechanism with four math problems and an open-source model. The model attempts each problem 100 times, producing a reasoning trajectory followed by a final answer. The system grades the attempts, increases the weight of the reasoning trace for correct answers, and discourages behavior from incorrect attempts. Repeating this process over batches teaches the model to solve the task. Applied Compute uses the same pattern for customer-specific work instead of mathematics. The example makes clear that the training system needs both sampling capacity for many attempts and a way to evaluate the resulting answers.
Enterprise RL runs need predictable completion times as well as high throughput
Applied Compute's runs differ from the large, multi-week training runs associated with frontier labs. The team wants to train a model and deliver it to a customer on the order of days. Runs also need to be cheap enough for the business to scale. Garg gives equal weight to a less obvious requirement: low variance in job duration. A run that is fast on average can still be difficult to operate if its completion time changes widely. This requirement motivates the systems work on GPU utilization, sampling and training overlap, and workload simulation.
Synchronous RL lets the slowest sample determine every training step
In synchronous RL, a batch of eight samples must finish before training begins. The next sampling phase starts only after the training step completes. GPUs that finish their requests early wait for the straggler, so step time is controlled by the longest sample. In an experiment with 40 arithmetic problems, 32 samples per problem, and Qwen 30B, 99 percent of samples finished in about 40 seconds, while the final percent took another 80 seconds. The throughput chart showed GPUs busy at the start and heavily underused near the end. Applied Compute calls this state of waiting GPUs 'slacking.'
Pipeline RL raises utilization by allowing training during sampling
Asynchronous RL removes the requirement that sampling and training happen in lock step. In the pipeline RL approach described by the speakers, some GPUs continuously sample while other GPUs train on completed samples pulled from a queue. After a training batch finishes, new weights are sent to sampling workers even when they are already generating requests. This in-flight update keeps the pipeline moving, but a single sample can contain tokens generated by several policy versions. The result is a sample with stale tokens, where different parts may be one, two, or more training steps behind the current policy.
More tolerated staleness reduces idle time but increases learning variance
The system can limit how stale a sample is allowed to become. If the limit is two steps, training may have to wait before sending another weight update because doing so would make an unfinished sample three steps stale. A limit of one creates even longer idle periods. Higher tolerated staleness usually means fewer idle GPUs, but the importance ratio used to correct policy-gradient training becomes more variable. Garg says that higher variance can make learning unstable and cause divergence. The practical trade-off is therefore between faster runs and the algorithmic work needed to make stale training data safe.
A simulator models GPU allocation before an expensive RL run
Li frames the end-to-end system as a modeling problem. The model includes the total GPU budget, the training batch size, sampling throughput, and training throughput per GPU. Sampling throughput is estimated from forward-pass latency as a function of batch size, with GPU memory and KV-cache capacity limiting how large the batch can become. Training throughput accounts for forward passes, backward passes, and optimizer work. The simulator also uses the response-length distribution, since longer responses affect both sampling time and the number of tokens processed in training. These inputs let the team compare layouts without first running the full job.
The asynchronous layout must balance production rate, consumption rate, and staleness
An asynchronous system can waste GPUs in either direction. With too many training GPUs, workers consume the queue faster than samplers produce work. With too many sampling GPUs, the queue grows and samples become increasingly stale. Li defines two constraints for selecting a layout. Sampling and training should produce and consume work at roughly equal rates. The simulated maximum staleness must also stay below the level that the learning method can tolerate. The simulator sweeps the number of training GPUs, infers the sampling allocation from the fixed compute pool, checks KV-cache limits, and removes configurations that exceed the staleness limit.
Simulation finds faster configurations without running every candidate
The end-to-end simulation estimates roughly a 60% speed-up against the synchronous baseline when compute is allocated well between training and sampling. The team can sweep layouts under memory and staleness constraints before spending money on a full RL run. This lets them study how the best allocation changes when response lengths become longer, which can happen as models learn to think for more tokens. It also gives them performance targets for systems optimization. Li says the simulator has informed both systems decisions and research design, including questions about GPU allocation and the throughput the stack should aim for.
"This allows us to ask answer scientific questions from first principles like what is the optimal configuration that we should have of our GPU compute if we made response lengths very long."19:29
Who should watch
You are building RL infrastructure for task-specific models and need runs that finish in days rather than weeks.
Your asynchronous training pipeline has idle GPUs, queue imbalance, or samples that become stale before training.
You want a workload simulator that can test GPU allocations and response-length assumptions before launching costly experiments.
