TL;DR
A decoding intervention borrows an RL model’s greedy token where it disagrees with its base model, while sampling from the base elsewhere. On 120 AIME problems, this recovers most of the observed RL pass@1 gain with about 4% of tokens replaced. Both models still run at every step: this is a diagnostic of useful decisions, not a compute-saving decoder.
Why it matters
RL’s own sampling distribution may obscure the value of its individual token preferences. The study separates opening cues, later context-dependent decisions and finite-budget coverage, showing why neither repetitive reflection words nor large-k accuracy alone fully explains the benefit of RLVR.
Key findings
- 01
The main pair is Qwen3-4B-Base and a GRPO checkpoint trained directly from it on DAPO-Math-17k without SFT. AIME 2022/2023 duplicates are removed from training; checkpoint selection uses a DAPO validation split. Training uses 128 prompts and eight rollouts per prompt, with reward +1 for a correct final answer and −1 otherwise.
- 02
At every decoding step, rl-replace compares the base and RL models’ greedy token IDs on a shared prefix. If they differ, it emits the RL token; otherwise it samples from the base at temperature 0.6 and top-p 0.95. Evaluation caps output at 8,192 tokens and grades final boxed answers with Math-Verify.
- 03
In the main 16-sample-per-problem run on AIME 2022–2025, pass@1 is 10.00% for the base, 18.44% for rl-replace and 19.32% for RL. The intervention recovers 90.5% of the RL-minus-base gain, with a 95% problem-bootstrap interval of 77.0–108.6%. About 4.0% of pooled generated tokens are replaced, or 4.5% when rates are averaged per response.
- 04
Forcing only the opening token “Alright” raises base pass@1 to 13.23%. Later replacements after that same opening add 5.21 absolute percentage points, with a paired 95% interval of 3.12–7.55. Fixed rules for inserting “Wait” or “Re” fail to reproduce this gain; the negative result is limited to the tested rules.
- 05
In a separate large-sample Qwen3-4B run, rl-replace reaches 67.50% pass@256 versus 61.40% for RL. The 6.10-point difference has a 95% interval of 0.82–11.02. Its 5.86-point advantage over the base has an interval of −0.33 to 10.71, so superiority to the base is less certain.
- 06
A whole-response mixture, choosing RL with probability 0.92, approximately matches rl-replace pass@1 at 19.00% versus 18.98%, but reaches only 61.45% pass@256. These comparisons equalize attempt counts, not compute. The large-sample run draws 512 responses per problem for base/RL and 256 for rl-replace, using an unbiased pass@k estimator.
- 07
MATH-500 tests extend the intervention to Qwen3-8B-Base and Llama-3.2-3B pairs, recovering 88.6% and 86.5% of their RL gains. Qwen3-8B also shows higher observed AIME pass@256 under replacement. Opening and reflective-token ablations remain specific to Qwen3-4B.
Evidence & limits
Sparse replacement is not sparse model execution: both models score every token. The authors report 7.3–7.9 H800 GPU-hours for one AIME year with 16 samples per problem and about 1,012 GPU-hours for the four-year, 256-sample intervention run. No compute-efficiency advantage is established.
An early replacement changes the prefix and can redirect every subsequent token. The approximately 4% rate is not a measure of an independent causal contribution, and opening and later-decision effects cannot be added as separate mechanisms.
Pass@k measures whether at least one of k attempts succeeds; it is not all-attempt reliability. Finite-budget coverage does not identify every reasoning path a base model can generate, so the experiment does not prove RL creates previously impossible reasoning paths.
The main benchmark has only 120 problems, with 30 per year. Problem-level bootstrap intervals quantify evaluation uncertainty, not variation across independently trained RL seeds. Independent evaluation runs vary, and the paper cautions against interpreting small cross-run differences.
The decoder requires compatible token IDs and access to an RL counterpart. A control using base-greedy tokens still consults the RL model to locate disagreements; it is not a fully base-only method.
Power sampling is tested with a particular MCMC configuration and output limit. Its failure to close the AIME gap does not rule out all inference-time search, learned intervention policies, or other compute budgets.
The detailed mechanism analysis concerns mathematical reasoning and one main model pair. The observed diversity/coverage association does not establish diversity loss as the cause of RL’s coverage behavior, nor a general scaling law.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-4B-Base, Qwen3-8B-Base, Llama-3.2-3B
- Algorithms
- GRPO, RLVR, rl-replace, Power sampling
- Benchmarks
- AIME 2022–2025, MATH-500
Topics