TL;DR
A controlled study of where reasoning diversity disappears during RLVR, distinguishing failure to initiate a solution family from failure to execute it after the first reasoning choice.
Why it matters
Higher pass@1 can hide a narrower policy. If training concentrates probability on fewer initial strategies, the model may look stronger on one sample while producing less value from additional test-time sampling.
Key findings
- 01
Across PPO and GRPO setups on the Countdown task, the authors report solution coverage falling by as much as 67% even as single-sample accuracy improves.
- 02
Likelihood shifts before the first arithmetic operation are reported to be 11–16× larger than shifts during downstream reasoning.
- 03
An entrance-targeted parameter intervention increases solution coverage by 37% with no reported loss in pass@1, while surface prompting does not recover the diversity.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-3B, Qwen2.5-3B-Instruct
- Algorithms
- RLVR, PPO, GRPO
- Benchmarks
- Countdown
Topics