TL;DR
RISE constructs a distillation teacher by extrapolating a model’s own RLVR training trajectory. Alternating reward-based updates with token-level distillation improves the tested reasoning and agent policies while reusing their sampled rollouts.
Why it matters
The source and update schedule of a teacher shape what distillation can contribute to RL. RISE makes the model’s recent learning trajectory a source of dense supervision, connecting teacher design, sample reuse, and reward quality.
Key findings
- 01
The method compares a current checkpoint with a trailing anchor and extrapolates their displacement in parameter or logit space. This creates a synthetic teacher without an external stronger model or privileged conditioning in the teacher prompt.
- 02
Each iteration has two phases: RLVR first provides a reward-grounded update, then on-policy distillation refines the student using the extrapolated teacher. The distillation phase reuses the RLVR rollouts, and the teacher is refreshed as the student changes.
- 03
Experiments span mathematical reasoning, mixed math and STEM, code generation, and multi-turn agent tasks with models from 1.7B to 8B. The authors report improvements over their RLVR-only and privileged self-distillation baselines across these task families.
- 04
Ablations support a contribution from both the RLVR and distillation phases. The useful extrapolation range narrows during training; the recipe reduces the extrapolation coefficient toward one and can smooth the reference checkpoint with an exponential moving average.
Evidence & limits
Reusing rollouts does not remove the cost of teacher scoring and distillation updates. In the reported configurations, each iteration takes about 1.3 times GRPO wall time on Qwen3-1.7B and 1.6 times on Qwen3-8B. These are iteration-time comparisons, not a general time-to-target speedup.
Extrapolation amplifies the direction of recent training, including mistakes in a hackable reward signal. The paper leaves principled detection of unreliable extrapolation open; empirical schedules do not establish that every extrapolated teacher is better.
The experiments are bounded training runs on 1.7B–8B models, not evidence of unbounded recursive improvement. Comparisons with fixed external-teacher pipelines require care: RISE uses a changing self-teacher and repeatedly alternates RLVR with distillation.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-8B, Qwen3-1.7B, Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, Qwen2.5-3B-Instruct, OLMo3-7B-Instruct-SFT
- Algorithms
- RISE, RLVR, On-Policy Distillation, GRPO, Logit-space extrapolation, Weight-space extrapolation
- Benchmarks
- GPQA-Diamond, IFEval, MMLU-Pro, ALFWorld, WebShop
Topics