TL;DR
On-policy distillation can prepare a student for later RL even when its immediate accuracy gain is small. Controlled downstream recipes show stronger reported final performance from OPD starting policies, and a distillation objective ranking that reverses after RL.
Why it matters
Choosing a preparation checkpoint solely by its current benchmark score can miss how well it will learn next. This study evaluates distillation through subsequent RL outcomes and highlights that trajectory source and divergence objective interact with later training.
Key findings
- 01
The text setting pairs Qwen3-8B with a Qwen3-32B teacher; the vision-language setting pairs Qwen3-VL-8B with Qwen3-VL-32B. Preparation uses 100K prompts per setting, followed by shared GRPO-style recipes on 17K text or 16K vision-language RL prompts.
- 02
Across eight equally weighted text benchmarks, the OPD starting score is 37.02 versus 36.48 for Base. After RL, reported averages are 59.18 for OPD, 51.90 for Base, and 47.48 for SFT initialization: OPD exceeds direct RL by 7.28 percentage points.
- 03
Across twenty vision-language benchmarks, OPD reaches 63.02 after RL versus 61.68 for Base and 61.33 for SFT. Diagnostics on 1,024 prompts with 64 responses each suggest pre-RL Pass@k alone does not explain subsequent performance.
- 04
For student-generated distillation prefixes, reverse KL leads forward KL before RL (60.66 versus 60.17), but forward KL leads afterward (64.87 versus 63.02). With teacher-generated prefixes, reverse KL remains ahead both before and after RL.
Evidence & limits
Shared downstream settings do not imply equal run lengths or total compute. SFT and OPD add preparation costs, so the reported final scores do not establish a compute-matched efficiency advantage.
Runs stop after qualitatively monitored training collapse, and the appendix states that the criterion for selecting reported checkpoints is unspecified in the available records. This limits interpretation of final-performance comparisons.
Each trajectory-source/objective variant has one distillation run and one RL run. Diagnostic bootstrap intervals quantify variation across prompts, not robustness across training seeds.
Teacher alignment and retained uncertainty are proposed explanations, not separately identified causal mechanisms. The probes use each policy’s own prefixes, mixing differences in visited contexts with conditional predictions.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-8B, Qwen3-32B, Qwen3-VL-8B, Qwen3-VL-32B
- Algorithms
- On-policy distillation, GRPO, Forward KL, Reverse KL
- Benchmarks
- Eight text benchmarks, Twenty vision-language benchmarks
Topics