Sizing the RL Fine-Tuning Loop: Rollouts, Verifiers and GPU Split

Overview

Reinforcement learning against verifiable rewards has become the standard way to improve reasoning and tool-use behaviour after supervised fine-tuning, and it breaks the mental model most teams carry from SFT. In supervised fine-tuning, GPUs do gradient steps on a fixed dataset. In an RL loop, the model generates candidate responses, a verifier scores them, and only then does a gradient step occur — and the generation phase typically dominates the compute. Sizing a cluster as though it were an SFT cluster produces hardware that is badly matched to the workload.

What the loop actually does

A single iteration proceeds in stages. Sample a batch of prompts from the task distribution. For each prompt, generate several candidate completions from the current policy — typically four to sixteen, sometimes more, because the learning signal comes from comparing outcomes across samples. Score each completion with a verifier, which for verifiable-reward settings is a deterministic check: does the code compile and pass tests, does the mathematical answer match, does the tool call succeed. Compute advantages from the scores and take a gradient step. Repeat.

The compute asymmetry is immediate. One gradient step consumes one forward and backward pass over a batch. Generating the rollouts for that batch consumes many full autoregressive generations, each potentially thousands of tokens long. In reasoning tasks where completions are long, generation can be an order of magnitude more expensive than the update.

Splitting the cluster

The practical consequence is that an RL post-training cluster looks like a serving deployment with a training node attached, not like a training cluster. That means inference-optimised software — continuous batching, paged attention, quantized generation — matters as much as training framework choice, and the generator fleet benefits from the same techniques described in disaggregated inference.

Weight synchronisation is the hidden cost

Generators must sample from the current policy, or close to it, so after each update the new weights have to reach every generator. For a large model this is gigabytes pushed across the cluster, repeatedly. Do it naively and the generators sit idle during every sync, destroying utilisation.

Three mitigations are standard. Allow slight off-policy drift, updating generators every few steps rather than every step, with an algorithm that tolerates it. Overlap synchronisation with generation so weights update while work continues. And keep generators and trainer within a fast fabric so the transfer is quick. Teams that skip this analysis frequently find their expensive generator fleet running at low utilisation for reasons that have nothing to do with the model.

The verifier is the real project

The appeal of verifiable rewards is that they avoid training a reward model, which is itself expensive and prone to being gamed. The cost is that you need a deterministic, automatable check for task success — and for most enterprise domains, that check does not exist yet.

Where it exists naturally, RL works well: code that must pass tests, mathematics with checkable answers, tool calls with observable success or failure, structured extraction validated against a schema. Where success is a matter of judgement — was this summary good, was this response appropriately empathetic — verifiable rewards do not apply and you are back to preference-based methods. The honest planning advice is to choose tasks where verification is possible before committing to an RL programme. The preference-based alternatives are covered in sizing DPO and RLVR post-training infrastructure.

Sizing for an enterprise programme

Work from experiment throughput rather than a token budget. A useful planning question is: how many complete RL runs do you need per month to make progress, and how long can each take? From there, estimate rollouts per run (prompts times samples times iterations), tokens per rollout, and required generation throughput. That gives the generator fleet. The trainer is sized by model memory as in any fine-tuning exercise.

Two India-specific notes. This workload is bursty and experiment-driven, which suits rented or shared capacity better than owned hardware for most enterprises — subsidised national compute at rates near a dollar per GPU-hour is well matched to campaign-shaped RL work. And the sandboxed execution environment for code verifiers needs care: running model-generated code requires proper isolation, and that is a security engineering task, not an ML one. The cluster fundamentals are in fine-tuning GPU server sizing.

When not to do this

RL fine-tuning is powerful and expensive, and it is frequently the wrong tool. If the goal is to make a model aware of enterprise knowledge, retrieval is cheaper and more maintainable. If the goal is to match a format or tone, supervised fine-tuning on a few thousand examples usually suffices. If the goal is to improve tool-calling reliability on a narrow set of tools, constrained decoding and better prompting often close most of the gap.

Reserve RL for cases where the desired behaviour is a capability rather than knowledge or format — multi-step reasoning, planning, recovering from failed tool calls — and where you have a verifier. That is a narrower set than the current enthusiasm suggests, and organisations that test the cheaper options first usually find they do not need the expensive one.

Frequently asked questions