When Do Larger Batches Help Scale LLM Reinforcement Learning?

TL;DR

An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.

Why it matters

A larger batch can make generation hardware more efficient while consuming more samples per update. This paper makes the trade-off operational: throughput gains have to exceed any samples-to-target penalty.

Key findings

1. 01

   With batch-dependent hyperparameters retuned, the authors find approximately batch-size-invariant sample-indexed learning over a bounded range.

2. 02

   Larger batches improve generation throughput by up to 2.29× on fixed hardware in the reported experiments.

3. 03

   For GRPO, the combined throughput gain and learning-rate retuning reduce time-to-target by up to 29%; simply increasing the batch without retuning is slower.

Scaling dimensions

Models, methods & benchmarks

Algorithms

GRPO, PPO, Adam

Topics