TL;DR
An empirical study of how to combine on-policy distillation with RLVR. Training with distillation first and RL second produces strong logic-reasoning gains, while the advantage over the strongest alternatives is less decisive on math.
Why it matters
Dense teacher feedback and sparse verifiable rewards can pull an update in different directions. Separating them into stages offers a practical way to expand a student’s solution coverage before optimizing how reliably it succeeds.
Key findings
- 01
The main experiments use Qwen3-8B in non-thinking mode as a frozen teacher and Qwen3-1.7B-Base as the student. Methods share the same total training-step budget across logic tasks and mathematical reasoning benchmarks.
- 02
OPD-then-RL reaches average pass@1 of 80.6 on the logic tasks, leading the other combined objectives by 11.7–26.7 percentage points. The study also evaluates a 0.6B Qwen student and an additional OLMo teacher–student pair.
- 03
Pass@k behavior, learning dynamics, and parameter-update analysis support the authors’ explanation: distillation expands coverage of teacher-supported solutions, while RL concentrates probability on successful solutions within that support. Mixing the signals in one update can create interference.
- 04
The OPD validation score is a useful signal for when to switch to RL. In the tested settings, an OPD cold start also works better than the compared SFT cold start, making stage allocation a concrete post-training design choice.
Evidence & limits
On math, OPD-then-RL has the highest reported average pass@1 at 31.8, but paired problem-level bootstrap tests find a statistical tie with the three strongest competitors. The evidence does not support claiming a significant win over every baseline.
Equal training-step budgets are not necessarily equal compute budgets: teacher scoring and the different training objectives have different costs. A controlled comparison with first improving the teacher using RLVR is left to future work.
The study focuses on stronger external, frozen teachers and a reverse-KL distillation objective. Self-distillation, multiple teachers, alternative distillation losses, and frontier-scale training may behave differently.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-8B, Qwen3-1.7B-Base, Qwen3-0.6B-Base, OLMo-3.1-32B-Instruct, OLMo-3-7B-Instruct-SFT
- Algorithms
- On-Policy Distillation, RLVR, GRPO, OPD-then-RL
- Benchmarks
- Knights & Knaves, Zebra Puzzles, Countdown, MATH-500, AMC23, AIME24, AIME25
Topics