Variance Reduction In Policy Gradient Estimation Via Baseline Subtraction

via “reinforce leave-one-out (rloo) policy gradient training”

Reinforcement learning from human feedback — SFT, DPO, PPO trainers for LLM alignment.

Unique: Implements leave-one-out baseline estimation with automatic variance monitoring and adaptive learning rate scaling, reducing gradient variance by 30-50% compared to standard REINFORCE without value function overhead

vs others: Lower variance than standard REINFORCE because it uses batch-level baselines; simpler than PPO because it avoids value head training and importance weighting; more efficient than GRPO for small batch sizes

### Other Papers <a name="2023op"></a>

Unique: Applies variance reduction techniques from actor-critic methods to language model policy gradients, enabling stable learning from high-variance trajectory data — this is distinct from vanilla policy gradient which can be unstable with sparse or noisy rewards

vs others: More stable than raw policy gradients because baseline subtraction reduces variance, and more sample-efficient than importance sampling because it doesn't require explicit off-policy correction