TL;DR
Score centering removes an accumulating drift term caused by differences between rollout and training engines. Experiments test when this additive correction stabilizes quantized or stale-policy RL, and when it benefits from being combined with importance sampling.
Why it matters
Faster rollout engines often introduce numerical or policy mismatch. Understanding which part of that mismatch destabilizes learning helps connect throughput improvements to a training procedure that can actually sustain them.
Key findings
- 01
The analysis separates a drift term from the reward-dependent update. When a biased sampler repeatedly receives the trainer’s weights, this drift can accumulate through a feedback loop; score centering subtracts the expected score to cancel it.
- 02
The experiments isolate corrections on a shared REINFORCE objective with group-centered rewards, using Qwen3-0.6B-Instruct on Countdown and Qwen3-30B-A3B-Base on INTELLECT-2 math.
- 03
With a severely quantized 30B sampler using INT8 and an INT4 KV cache, score centering reaches 30% training accuracy versus 12% for token-level importance sampling, while the other compared methods finish below 5%. These are training accuracies under the specified stress test.
- 04
An approximation using the sampler’s top-k log probabilities avoids requiring the full next-token distribution. Under severe policy staleness, composing score centering with importance sampling works better than score centering alone.
Evidence & limits
The headline comparisons deliberately amplify mismatch on short sequences to separate methods within affordable runs. They are a proxy for longer training with milder mismatch, not a direct demonstration of equivalent gains in production-scale long-context training.
Removing drift does not fully correct the sampling distribution. Severe staleness remains a reason to combine the method with importance sampling rather than discard that correction.
The reported 30B results measure training accuracy under particular quantization settings. They do not establish a general held-out capability gain or an end-to-end speedup.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-0.6B-Instruct, Qwen3-30B-A3B-Base
- Algorithms
- Score centering, REINFORCE, Importance sampling
- Benchmarks
- Countdown, INTELLECT-2 math
Topics