TL;DR
GRPODropout selectively excludes high-probability positive-advantage rollouts from GRPO updates, then recenters retained advantages using old-policy probability weights. With the same generation budget, experiments on Qwen3 models report higher average benchmark accuracy and less entropy collapse than standard GRPO.
Why it matters
Rollout usage is a separate lever from generating more samples or changing the reward. This study shows why selectively dropping successful responses needs a corresponding advantage correction, and why fewer update samples should not be confused with lower rollout-generation cost.
Key findings
- 01
The method estimates rollout entropy contributions from advantages and old-policy log-probabilities, greedily removes negative-contribution candidates, restores all nonpositive-advantage rollouts, and recenters the retained advantages before applying the original loss. The implementation length-normalizes log-probabilities for filtering and recentering.
- 02
Experiments train Qwen3-1.7B, 4B, and 8B for one epoch on DAPO-17K. Each batch starts with 128 prompts and eight responses per prompt: 1,024 generated candidate rollouts before filtering. The response limit is 8,192 tokens.
- 03
The unweighted average across ten benchmarks rises from 54.13 to 55.01 for 1.7B, 63.03 to 65.85 for 4B, and 66.33 to 67.29 for 8B relative to GRPO. These are gains of 0.88, 2.82, and 0.96 percentage points, averaging 1.55 points across model sizes.
- 04
The main evaluation uses Mean@32 for AMC23, AIME24, AIME25, and HMMT25, averaging correctness across 32 responses, and Mean@1 for the other six benchmarks. Mean@32 is not success on at least one of 32 attempts.
- 05
For Qwen3-4B, ten-checkpoint measurements excluding all-zero-advantage groups retain 7.03 of eight responses per prompt on average, or 12.1% fewer retained update rollouts. The candidate generation budget remains unchanged.
- 06
Recentering is critical in the reported ablations: removing positive rollouts without recentering yields an average score of 16.00 for 4B versus 65.85 with the full method, and 28.81 versus 55.01 for 1.7B.
- 07
A separate Qwen3-4B diversity analysis on 130 math questions reports Pass@32 of 75.21 versus GRPO’s 63.54; Clip-Cov reaches 76.88. This distinct experiment uses a success-at-least-once metric, unlike the main table’s Mean@32.
Evidence & limits
The same rollout generation budget is used. Fewer retained update rollouts do not establish a corresponding reduction in wall-clock time, GPU cost, or generated tokens.
GRPODropout does not lead every baseline at every scale: KL-Cov scores 66.13 versus 65.85 at 4B and 67.32 versus 67.29 at 8B. Some individual general-knowledge benchmark scores also regress relative to GRPO.
The theoretical results rely on local update and auxiliary model assumptions; they motivate the heuristic rather than guaranteeing global entropy preservation or better accuracy throughout neural-network training.
The study uses math training on three Qwen3 sizes with eight responses per prompt. Single-rollout use, very small groups, coding, and long-horizon agent training are not established by these experiments, and the main table does not establish multi-training-seed robustness.
Reasoning-path diversity relies on GLM-5.2 summaries and bge-m3 embeddings. Its shared easy/hard partitions exclude 28 mixed-membership questions, and embedding diversity is not proof of correct or novel reasoning.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-1.7B, Qwen3-4B, Qwen3-8B
- Algorithms
- GRPODropout, GRPO, Probability-weighted advantage recentering
- Benchmarks
- GSM8K, MATH500, AMC23, OlympiadBench, AIME24, AIME25, HMMT25, ARC-C, MMLU-Pro, SuperGPQA
Topics