TL;DR
ProbePO spends a capped continuation budget on selected reasoning prefixes, then uses their estimated values to redistribute GRPO credit across tokens. It preserves each response’s mean advantage and keeps probe continuations out of the actor batch. Mathematical-reasoning experiments report accuracy gains, with separate comparisons against GRPO and dense probing.
Why it matters
Fine-grained credit assignment need not require a learned critic or probing every step. ProbePO treats prefix evaluation as a limited measurement budget and makes the resulting training costs explicit, while showing why noisy value estimates need regularization.
Key findings
- 01
The selector ranks reasoning boundaries by the maximum full-vocabulary entropy over an eight-token forward window on the generated trajectory. Selection uses no extra reward queries, but selected prefixes are evaluated with verifier-scored continuations. The method as a whole still requires rewards.
- 02
The headline recipe generates eight backbone responses per prompt and allocates at most 40 auxiliary continuations across the group: up to five pivots with eight continuations each. Prefix values are shrunk toward the group reward mean with pseudo-count four. A centered segment residual with weight one redistributes credit while preserving each trajectory’s GRPO token-mean advantage.
- 03
Probe continuations are measurements only; they are not added to the actor training batch. Zero pivots recover the GRPO advantage under matched implementation settings. Mean-advantage preservation is an algebraic property, not a claim that nonzero-pivot policy gradients are unchanged.
- 04
Terminal six-benchmark averages are 19.34% versus 17.62% for GRPO on DeepSeekMath-7B-SFT, 48.37% versus 46.49% on R1-Distill-Qwen-1.5B, and 41.11% versus 38.26% on Qwen3-4B-Base. The absolute gains are 1.72, 1.88 and 2.85 percentage points. ProbePO leads GRPO in all 18 model–benchmark cells and leads the four-method comparison in 16.
- 05
The reported Mean@16 scores average sampled-answer correctness, not best-of-16 success; AIME24 uses Mean@32. Avg-6 weights the six benchmark scores equally. Terminal checkpoints are fixed at 1,000 updates for DeepSeekMath/R1 and 27 for Qwen3 rather than selected separately for each test benchmark.
- 06
In the separate DeepSeekMath dense-probing comparison, ProbePO reports 231 H200-hours against VinePPO’s 944, with Avg-6 of 19.34% versus 17.31%. The approximately 76% time reduction is relative to this VinePPO setup. Both use eight H200 GPUs and 1,000 updates; ProbePO’s at-most-2,560 auxiliary continuations per update are a cap, whereas VinePPO’s 27,265 are an observed mean.
- 07
A separate equal-time Qwen3 study on eight B200 GPUs reports MATH-500 of 80.01% versus 75.52% and Avg-6 of 41.03% versus 39.08% after about 11.14 trainer hours. Timing includes backbone/probe generation, optimization, evaluation and checkpointing; these are not the headline terminal-table checkpoints.
Evidence & limits
The dense-probing speedup is not a 76% saving over GRPO. Hardware, response limits, graders and recipes define its scope, and the study’s VinePPO score uses a stricter grader than the original VinePPO report.
The three model families use different data partitions and prompts. DeepSeekMath/R1 use an 11,500-problem PRM800K/MATH repartition that includes original MATH test problems outside the held-out set; Qwen3 uses the original 7,500-problem training split. Scores support within-model comparisons, not clean cross-family scaling conclusions.
The R1 setup has a binding 2K context and uses cutoff rescue for unfinished probes. In recipe-matched replay, 2% need cutoff handling; those values estimate the modified rescue continuation policy rather than the unmodified actor policy.
The random-placement ablation replaces the entire structured placement module, including boundary and spacing rules, with uniform interior-token placement. It does not isolate entropy ranking alone. The shrinkage ablation separately tests removal of the baseline pseudo-count.
The main accuracy tables provide point estimates without independent-training-seed uncertainty. Many sampled evaluation answers are not a substitute for repeated training runs, and small per-benchmark differences should be interpreted cautiously.
Token-mean preservation does not establish unbiased token-level causal credit. Sparse continuation estimates, shrinkage bias, segment boundaries and policy clipping affect updates; the theoretical allocation benchmark relies on conditional approximations.
Experiments cover mathematical reasoning in three 1.5B–7B backbones. They do not establish the same gains for coding, open-ended agents, larger models, or a general compute scaling law.
Scaling dimensions
Models, methods & benchmarks
- Models
- DeepSeekMath-7B-SFT, R1-Distill-Qwen-1.5B, Qwen3-4B-Base
- Algorithms
- ProbePO, GRPO, VinePPO, SPO-tree, TreePO
- Benchmarks
- MATH-500, OlympiadBench, CollegeMath, AMC, MinervaMath, AIME24
Topics