RL is Everything, Everywhere, All at Once

Reinforcement learning has become the standard final stage of training a language model. What began as RLHF for aligning chat models is now, since DeepSeek-R1 popularized GRPO on verifiable rewards, the standard recipe for frontier agentic models such as Kimi K3, Cursor's Composer 2, and Cognition's SWE-1.7.

A modern RL pipeline is everything, everywhere, all at once: every major workload in one system, spread across whatever compute you can get, all coupled in one feedback loop. A single run involves:

• Inference — rollout engines serving hundreds to thousands (§5.3.2) of concurrent agent loops, and often judge models grading finished rollouts
• Sandboxes — isolated environments executing each agent's actions
• Training — GPUs consuming the samples and computing policy updates

Spanning all of these are two more demands:

• Weight sync — the trainer's new weights must reach the engines after every step
• Reliability — the whole loop must hold together for weeks (§3) at a time

As the Miles team puts it: it is a distributed systems problem. Put together, a representative system looks like this:

This system raises infrastructure problems and tradeoffs at every layer, including scheduling, placement, weight transport, throughput, isolation, and recovery.

Below, we walk through the life of an RL pipeline and discuss each in turn.

Same everything, many architectures #

Even with the standardized components above, teams face several choices for how to run their RL pipeline. The most common knobs are colocation versus disaggregation, synchronous versus asynchronous stepping, and the choice of weight-sync transport.

Frontier systems sit at every point on this spectrum.

Kimi K3 colocates training and rollout on the same GPUs, offloading training state to NVMe between phases (§5.3.1). This reduces weight sync to an on-device operation, at the cost of every GPU time-slicing between roles.

slime's default mode disaggregates them onto separate pools, broadcasting weights between them over NCCL. This lets training and inference scale independently, while fast weight sync benefits from keeping both pools on a shared high-bandwidth fabric.

Cursor's Composer 2 and Cognition's SWE-1.7 both ran asynchronous pipelines spanning multiple regions. This maximizes rollout throughput and availability, at the cost of policy staleness.

In fully asynchronous pipelines, generation and training run continuously. The trainer pushes weights to the object store, and each inference engine pulls them on its own clock.

These architectural choices and tradeoffs will be discussed in further detail as we move through the pipeline.

Place everything everywhere, except the trainer #

The architecture chosen determines how teams are able to place the various components of RL.

Colocation guarantees that the trainer and inference engines are scheduled as a single unit on the same set of GPUs, but trades away heterogeneous hardware, independent scaling, and asynchrony.

Disaggregation lets each component be placed on independent, heterogeneous hardware.

In practice, colocation is an easy way to start, since it only needs one pool of GPUs to schedule and no weight transport over network.

Disaggregation becomes attractive at scale, for two reasons:

1. Each component can run on hardware that suits it — a trainer requires compute-and-interconnect-dense GPUs, while inference can run as multiple replicas on cheaper, bandwidth-oriented hardware, and sandbox fleets need large pools of CPU.
2. Separate pools unlock asynchronous execution, one of the biggest throughput levers for large runs, covered below.

Regardless of the architecture, trainer ranks almost always sit on a single fast fabric: they exchange activations and gradients continuously, at volumes only NVLink or InfiniBand can sustain.

The “Gang” starts all at once, or not at all #

When disaggregated systems request components on heterogeneous resources, the scheduling complexity of the system compounds.

An RL step only completes if every component is up — a trainer admitted without its rollout engines or sandboxes burns GPU hours polling an empty queue, and two half-admitted runs can deadlock a cluster.

These resources often sit behind separate queue and quota systems, which don't know about each other and therefore can't guarantee gang scheduling. This problem is exacerbated across clusters, where no centralized scheduler exists.

Workarounds often mean custom coordination logic, such as polling, retries, and manual sequencing across schedulers. Even then, components hold GPUs idle while waiting for the rest of the loop to start.

Millions of agents need isolated environments #

Once the components have been scheduled and placed, the pipeline runs agent rollouts to produce training data, and every agent needs an isolated environment to act in.

The industry has converged on sandboxes as that environment layer, especially for coding agents running untrusted, model-written code: Moonshot built AgentENV (§5.3.2) for Kimi K3, Cursor runs rollouts on Anyrun (§6.2), and Anthropic has discussed spending over $1B on RL environments.

A sandbox is more than a container running bash: modern pipelines lean on snapshot and fork to judge finished rollouts without side effects (§5.3.2), and on egress control (§6.2) over what an agent's actions can reach.

The fleets can be enormous: Kimi K3's training and evaluation created 51 million sandboxes (§5.3.2), launched in bursts of tens of thousands within seconds — and at any moment most sit paused waiting on inference, as much as 98% (§5.3.2) of a sandbox's lifetime.

Chasing throughput trades away training stability #

This many interdependent components multiplies the surface area for bottlenecks, which compound on pipelines that last weeks. Generation, training, and weight sync are the obvious suspects, but slowdowns can surface almost anywhere in the pipeline.

One of the most common strategies for maximizing throughput is asynchronous execution.

Synchronous pipelines run each stage in turn, so each step takes as long as all stages combined.

In synchronous training, each “Step” waits for generation, training, and weight synchronization to complete before moving forward. The model is always trained on samples generated from its most recent version.

Asynchronous RL runs the whole loop all at once by overlapping each stage continuously, but allows the policy generating samples to lag behind the policy being trained. This lag comes with measurable costs to training stability, which compound when step throughputs drift too far apart.

Asynchronous RL systems overlap training with generation. In this illustration, samples arrive twice as fast as the trainer consumes them: v4 samples are already waiting while the trainer still consumes v2.

Asynchronous systems often pair with mitigations for staleness, from capping how stale a sample may get (§5.1) to dropping the stalest samples from the trajectory queue, a design space we'll dig into in a follow-up post. Done right, the tradeoff pays off — AReaL reports up to 2.77× faster training with matched or even improved final performance versus synchronous systems on the same GPUs.

Not every bottleneck needs that much machinery, though: in our slime example, scaling the inference fleet with Job Groups nearly halved RL step time. We tracked down the bottleneck by tracing slime's metric for pending inference requests. Sometimes the fix is boring — find the lagging component and scale it.

Weight synchronization trades speed for reach #

After every optimizer step, the trainer produces new weights, and must transport them to the rollout engines in order to continue generating training samples from the most up-to-date model.

A variety of transport methods exist, posing a tradeoff between speed and placement flexibility:

Four major transport methods used in industry. Each row down the table trades off speed for wider placement flexibility.

Frameworks such as slime support the first three out of the box. Object-store transport needs support in the RL framework itself.

Various methods, such as compression and delta-only transmission, exist for reducing the size of the weights transported each step: Cursor's per-step deltas compress to a handful of gigabytes for its 1T-parameter model (§6.2), synced through S3 to inference clusters across the US and Europe.

The run must outlive GPU failures #

Everything so far assumes the machines stay up. At scale, hardware failure is routine: Meta logged an unexpected interruption every ~3 hours (§3.3.4) during Llama 3 pretraining, 78% of them hardware. RL runs live on the same machines for weeks, with more moving parts.

The blast radius depends on what component failed. A dead rollout engine costs throughput, but the rest of the fleet can keep generating. A dead trainer rank stalls the entire training process. The replacement must recover and rejoin the trainer's fabric under the same placement constraints as the rank it lost.

Recovery therefore has to work at both the hardware and the software layer — detect the faulty machine, replace it, and resume the group while minimizing interruptions to the workload.

One platform for the loop #

At SkyPilot, we're building a single platform to run everything, everywhere, all at once — on your own compute, across clusters and clouds. Each problem in this post maps to a capability of the platform:

All of it launches as one file:

sky jobs launch rl.yaml

And we're building ahead — stronger gang guarantees across queues and clusters, and Endpoints as the rollout-inference layer inside Job Groups are both in progress.

If you're training frontier models on your own compute we'd love to build with you. Run the slime example → or talk to us →