Beyond SFT: Sizing DPO and RLVR Post-Training Infrastructure
Overview
Supervised fine-tuning stopped being the whole story: the standard 2026 recipe for producing capable, task-specialised models is a three-stage pipeline — SFT for format, preference optimisation (DPO-family) for alignment, then reinforcement learning with verifiable rewards (GRPO/DAPO-class) for reasoning — a stack surveyed in references like LLM-Stats’ post-training review. The infrastructure news is that stages two and three are heavier and differently shaped than SFT: DPO doubles model copies in memory, and RL adds a rollout (generation) workload that can dwarf the training itself. This article sizes the full pipeline for teams graduating beyond LoRA-only fine-tuning.
Key takeaways
- The 2026 recipe is SFT → DPO → RLVR; each stage has a distinct memory and compute signature, and skipping straight to RL wastes compute on format-learning.
- DPO holds policy and reference models simultaneously — roughly double SFT’s memory at the same model size; PEFT variants soften this.
- GRPO removed PPO’s critic network, roughly halving RL memory overhead — the reason verifiable-reward RL now runs at hobbyist-to-enterprise scales.
- RL is rollout-dominated: most GPU time goes to generating candidate answers, so an inference-optimised serving tier beside the trainer is the real architecture.
- Verifiable rewards (unit tests, math checkers, tool-call validation) make RL practical for enterprise tasks — the reward harness is engineering, not GPU spend.
The pipeline, stage by stage
SFT teaches structure — response format, tool-call syntax, domain register — and follows the memory math in our LoRA/QLoRA guide. Preference optimisation (DPO/SimPO-family) teaches taste: it trains on chosen-versus-rejected pairs and holds two model copies — the training policy and a frozen reference — making an FP16 full-parameter DPO of a 13B model an ~8×A100-class job, while LoRA-DPO of the same model fits a 96 GB workstation card. RLVR teaches reasoning: the model generates multiple candidate solutions per prompt (rollouts), an automatic verifier scores them, and GRPO-class updates push toward verified success — the recipe DeepSeek-R1 operationalised and the open ecosystem (TRL v1.0’s unified stack, NeMo RL, verl-class frameworks) has since commoditised, per practical GRPO guides.
Why GRPO changed the hardware math
PPO — the RLHF workhorse of 2023–24 — required four model instances (policy, reference, reward model, critic), pricing RL out of most organisations. GRPO computes advantages from groups of rollouts, eliminating the critic entirely and roughly halving per-step memory at comparable quality. Combined with PEFT and quantised references, the floor dropped dramatically: single-GPU GRPO on 7–8B models is now routine in TRL, a 96 GB QUASAR-class card handles 13–32B experiments, and serious multi-node RL begins where model size and rollout throughput demand it — not as an entry fee. This is the same democratisation arc QLoRA delivered for SFT in 2023, now replayed for reasoning training.
The real architecture: trainer plus rollout farm
RL post-training is generation-dominated: producing 8–64 candidate answers per prompt, with long reasoning chains, consumes the majority of GPU time. Production frameworks therefore split the estate — a training partition (bandwidth-heavy, holds optimiser state) and a rollout partition running an inference engine (vLLM-class), exchanging weights and trajectories over at least 100 GbE-class links per current framework guidance. Two sizing consequences. First, your inference-serving skills and hardware transfer directly: rollout nodes are spec’d like the serving nodes in our inference checklist. Second, rollout demand scales with reasoning length — training a model to think longer makes the next epoch more expensive, a feedback loop that budget owners should see coming.
Rewards: where engineering replaces GPUs
RLVR works where correctness is checkable: code (unit tests), maths (verifiers), structured extraction (schema validation), tool use (execution success) — which maps remarkably well onto Indian enterprise use cases like reconciliation, document processing and agentic workflows. The reward harness — sandboxed executors, test suites, validators — is CPU-side engineering that determines whether RL improves your model or teaches it to game the checker. Budget it as a first-class workstream. Subjective tasks still route through preference data and reward models, at correspondingly higher data cost. Teams should also hold out contamination-free evaluation sets from day one; the fastest way to waste an RL budget is optimising against a leaky benchmark.
Stage-to-hardware map
| Stage | 7–8B model | 13–32B | 70B+ |
|---|---|---|---|
| SFT (LoRA/QLoRA) | 24 GB card | 48–96 GB card | 96 GB (QLoRA) / multi-GPU |
| DPO (PEFT) | 48 GB card | 96 GB card | 2–8 GPU node |
| DPO (full-parameter) | 4–8 GPU node | 8 GPU node | Multi-node |
| GRPO/RLVR (PEFT) | 1–2× 96 GB or 4-GPU node | 4–8 GPU node + rollout GPUs | Multi-node trainer + rollout farm |
| Rollout serving | Shares trainer at small scale | Dedicated inference GPUs | Dedicated partition, 100 GbE+ to trainer |
Frequently asked questions
Do enterprises actually need RL post-training?
For format and tone, no — SFT and DPO suffice. RLVR earns its cost when the task has checkable correctness and residual error is expensive: code generation, structured extraction, agentic tool use. Start with SFT+DPO and add RL where verifiable gaps persist.
What is the minimum hardware for GRPO experiments?
A single 24–48 GB GPU runs TRL’s GRPO on 7–8B models with PEFT; a 96 GB workstation card covers 13–32B experimentation. Production runs with serious rollout volume want a 4–8 GPU node or a trainer-plus-rollout split.
Why does DPO need twice the memory of SFT?
It scores outputs against a frozen reference model while training the policy — two full model copies resident. PEFT adapters and quantised references reduce the footprint substantially at small quality cost.
Where does the GPU time actually go in RLVR?
Mostly to rollouts — generating and scoring many candidate answers per prompt, with long reasoning chains. Plan the rollout partition like an inference fleet; the gradient updates themselves are the smaller line.
Can post-training run on rented capacity?
Well — the pipeline is bursty and checkpointable, suiting IndiaAI-pool or commercial rentals. Keep preference and reward data (often your most sensitive IP) on infrastructure under your control, and repatriate checkpoints per your data posture.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.