TL;DR
A GRPO replay method that reuses past rollout groups according to their remaining learning value and compatibility with the current policy. Experiments examine when replay can replace costly fresh environment interactions in reasoning and agent training.
Why it matters
The next useful training sample may already be in the buffer. This paper connects rollout scaling to a practical decision: whether to generate more experience or spend compute selecting and reusing experience the policy can still learn from.
Key findings
- 01
Headroom prioritizes stored groups with remaining learning value; Drift filters groups that have moved too far from the current policy. The method preserves the fresh on-policy stream and adds no auxiliary generation pipeline.
- 02
On mathematical reasoning, the authors report the highest average Mean@32 across their full baseline set, including larger on-policy batches, naive replay, DAPO, ExGRPO, and BAPO. Mean@32 averages the scores of 32 sampled responses per problem.
- 03
On Search-R1, Headroom-Drift in the b128+r64 configuration achieves average Mean@32 of 0.3577 at 166.3 seconds per step. The larger fresh-rollout baseline (b192) scores 0.3212 and takes 197.2 seconds per step.
- 04
A matched 7B Search-R1 comparison improves average Mean@4 from 0.3737 with naive replay to 0.3955 with Headroom-Drift. This is a separate scale check with four samples per input, rather than the main evaluation with 32 samples.
Evidence & limits
Against matched naive replay in the main Agentic Search experiment, the average Mean@32 gain is only 0.0029 (0.3577 versus 0.3548), which the authors describe as within the noise range. The larger Best@32 gain measures the best sampled response, not typical single-response quality.
The timing result is a comparison of mean step times in a specific Search-R1 setup, not a demonstrated reduction in total cost to reach a fixed quality target. Selection adds overhead: matched naive replay takes 154.4 seconds per step, less than Headroom-Drift.
The 7B scale check has one training run per method and uses an 8-bit optimizer. These results do not establish a general scaling law or guarantee the same gains for frontier models and other environment distributions.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-Math-1.5B, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct
- Algorithms
- GRPO, Headroom-Drift Replay, DAPO
- Benchmarks
- AIME24, AMC23, MATH-500, Minerva, OlympiadBench, Search-R1 evaluation suite, Geometry3K, MathVista, MathVision
Topics