A step of an agentic RL run spends far longer generating rollouts than it spends training on them. So for most of every step, the training GPUs sit idle while generation requests back up on a single inference engine.
This isn’t specific to one setup: across recent large-scale RL systems, rollout generation is routinely the single largest wall-clock component of a step (a growing pile of systems papers all land here). So adding trainer GPUs won’t help, because the trainer already finishes early and then waits.
If the bottleneck is inference, that’s the thing to scale, independently of the trainer. This post walks through doing exactly that with slime, an RL post-training framework, on SkyPilot Job Groups. We train a Qwen3-14B coding agent on real bug-fixing tasks and scale the inference fleet from one SGLang engine to three. Async step time falls by almost 50%, from 1200s to 661s, and the trainer config never changes across the whole sweep.
Why agentic RL is inference-bound #
A year or two ago, RL post-training mostly meant RLHF on preference pairs. Now a lot of it is verifiable rewards on coding tasks: hand the model a bug, let it produce a fix, and score that fix automatically. Meta’s SWE-RL did this in a single shot, rewarding a patch by its textual similarity to the human fix. The newer work is more agentic: the model acts over many turns, and the reward is a test suite that passes or fails. Last year, Berkeley’s Agentica and Together AI trained DeepSWE this way, and Z.ai’s GLM-4.5 reported running its agentic RL disaggregated and asynchronous, on slime. The largest open models are built the same way: Moonshot’s Kimi K3, the 2.8-trillion-parameter model released in July 2026, was trained with agentic RL over asynchronous, large-scale rollouts, and Moonshot open-sourced AgentENV, a Firecracker-microVM sandbox platform, just to run those rollouts at scale.
The infrastructure gets complicated because of the shape of an agentic rollout. Every rollout is a multi-turn loop: the agent reads code, generates an edit, runs a tool, reads the result, generates again. Every turn is a generation request to an inference engine, and in the meantime the agent is blocked on a sandbox running tests. So most of the rollout is due to inference latency, notably with a long tail as some rollouts take many more turns than others. A batch only finishes when its slowest rollout does, so a few “stragglers” can leave hundreds of trainer GPUs sitting idle. In one teardown of production RL runs, SemiAnalysis found a GLM-model run with frequent tool use left the trainer idle 74% of the wall-clock time, its GPUs able to consume samples 5x faster than the inference fleet could produce them.
Most RL frameworks try to address this issue in the same way: put rollout and training on separate GPU pools, connect them through a rollout buffer, and sync weights asynchronously. Hugging Face surveyed 16 RL libraries and found the field had largely converged on it.
AReaL, for instance, reports a 2.77x speedup just from decoupling generation from training. slime is the SGLang-native option in this group, and the framework Z.ai used to ship the GLM model line.
Disaggregation is the right architecture, but it solves one problem (utilization) while introducing another one: orchestration. You no longer have a single job. You have a trainer job and N inference jobs that all have to come up together, find each other, and share weights, and you want to change N without touching the trainer.
That’s a different shape than the usual “one job, K identical replicas,” and it needs a different abstraction: something that treats a set of heterogeneous jobs as one unit you can scale component by component. SkyPilot Job Groups are that abstraction, and the rest of this post is what they let you do.
The abstraction: disaggregated jobs you scale independently #
slime runs a Megatron trainer and one or more external SGLang inference engines. A router in front of the engines speaks the OpenAI API, so the rollout code generates against it like any other endpoint. After each optimizer step the trainer publishes new weights and the engines reload them.
A Job Group lets you treat the trainer and the engines as a single entity. The jobs are gang-scheduled, so the trainer and all N engines come up, run, and tear down together: you never end up with a live trainer waiting on engines that failed to launch. Each job also gets a stable hostname (e.g. sglang-0.<group>), so the trainer’s router knows where to send rollout traffic and weight-sync calls without standing up a service registry. And the jobs don’t have to share hardware: this example runs H100s everywhere, but you could move inference onto cheaper GPUs by editing resources.accelerators on the engine job.
The training task is bug-fixing from SWE-smith, a dataset of real repository bugs. Each rollout drives mini-swe-agent through a read-edit-test loop. The agent’s code runs in a SkyPilot Sandbox, an isolated pod, so a bad edit can’t reach the cluster, and the reward function is simple: reapply the diff over the hidden tests, run them, and give reward 1 if they pass.
Scaling the engines instead of the trainer #
We ran the example on our own H100s and used slime’s built-in metrics to watch the bottleneck. With one engine, each step is mostly rollout, and that rollout is inference-bound: generation requests stack up on the single engine faster than it can drain them. Here’s the queue depth over two steps, one character per time bucket, peaking at 122 waiting requests:
That queue is the trainer’s idle time, drawn out over the step. Adding engines drains it: same Job Group and same trainer; we just add SGLang jobs and their names to the trainer’s engine list.
The plot below is the whole result. We ran the sweep in two modes. In async mode, training overlaps the next rollout, so step time roughly tracks rollout time; in sync mode, each step runs rollout then train, so step time is roughly the two added together. Either way, adding engines is what moves the curve:
Going from one engine to three cuts async step time from 1200s to 661s (about 1.8x) and takes the median queue to zero. By three engines, inference is keeping up with the rollout, so the queue sits empty for most of the run and peak depth drops from over 120 waiting requests to single digits.
Notably, the trainer configuration is identical across all six runs. The only thing we changed was how many SGLang jobs were in the Job Group. When the bottleneck is inference throughput, that’s exactly what you want to be able to do: add inference capacity and leave the trainer as-is.
These are single two-step runs, so treat them as the shape of the scaling curve rather than precise multipliers.
Run it #
You need a Kubernetes cluster with 5 to 7 free H100s (the trainer takes 4, each engine takes 1), a ReadWriteMany storage class for weight sync, and Sandboxes enabled on your API server.
First create the shared volume the trainer and engines use to pass weights, then launch:
Scaling the fleet is a different YAML with more engine jobs:
Launching the three-engine group brings up the trainer and all three SGLang engines together; the trainer’s rollout and loss metrics stream in the terminal as the run gets going:
The manifest itself is one file: an sglang task and a trainer task under an execution: parallel Job Group header, with primary_tasks: [trainer] so the group ends when training does. Each extra engine is just another identical SGLang job in the manifest, plus its name in the trainer’s SGLANG_MEMBERS list. All three manifests (coding-agent.yaml plus the two- and three-engine variants) live in the docs example.
The defaults are tuned for this run, so you don’t have to touch anything to reproduce it. When you do want to tune — async vs sync, the weight-transport path, the concurrent-rollout and warm-sandbox counts — that same docs example documents every trainer env var.
What could be improved even further #
Two things we’d improve with more time. The inference fleet is fixed for the duration of a run right now; with long, variable-latency rollouts it should really autoscale with demand instead. And weight sync goes through a shared volume, which pins the trainer and engines to the same cluster: move that sync onto an object store and you could split inference and training across clusters, or across clouds.
The main point still stands: the bottleneck in agentic RL is inference, not training, so the thing you want to scale is inference. The reason to reach for Job Groups here is that they turn “disaggregated and async” into a manifest you can declare, rather than a pile of manually-run scripts.
The full write-up (every manifest, the rollout code, and the benchmark data) is in the SkyPilot docs example.
To receive latest updates, please star and watch the project’s GitHub repo, follow @skypilot_org, or join the SkyPilot community Slack.
