TL;DR
An empirical study of how prompt difficulty and reasoning length affect on-policy distillation. Eight selected hard math problems approach the performance of a 17K-problem baseline, while repeated rollout generation and dense teacher supervision remain part of training.
Why it matters
Unique prompt count is only one dimension of a distillation dataset. This study separates prompt diversity from the reasoning trajectories those prompts generate, helping frame which data and compute budgets should be compared.
Key findings
- 01
The primary teacher–student pair is JustRL-DeepSeek-1.5B and DeepSeek-R1-Distill-Qwen-1.5B. The authors rank a 1,000-problem pool using both models’ accuracy over 16 rollouts per problem, then compare easy, medium, and hard prompts.
- 02
In the main mathematical reasoning comparison, eight hard prompts achieve 53.6% average accuracy versus 53.7% for the DAPO-Math-17K baseline. Increasing the selected hard set to 16 or 64 prompts yields no further gains in this setup.
- 03
Length-controlled experiments weaken the explanation based on high token entropy alone. Longer reasoning trajectories from hard prompts maintain teacher–student alignment over longer prefixes and expose reasoning patterns such as reflection and alternative solution attempts.
- 04
Additional teacher–student pairs span models up to 7B. Eight hard prompts reach 59.5% versus a 59.6% full-set baseline for the DeepSeek 7B student; the Qwen3 1.7B and 4B students reach 21.3% and 29.2%, compared with full-set baselines of 22.5% and 30.8%.
Evidence & limits
Eight prompts means eight distinct training problems, not eight total trajectories. The default comparison uses 279 training steps, batch size 64, and eight responses per prompt, with repeated on-policy generation and teacher scoring. The result does not establish a proportional reduction in compute.
Selecting hard prompts also consumes resources: the difficulty-ranking stage samples from both student and teacher over a 1,000-problem pool. Prompt counts omit this selection cost as well as the token cost of longer reasoning paths.
The evidence centers on mathematical reasoning with 1.5B–7B students. The selection rule is empirical, and some model pairs remain below their full-set baselines; the study does not establish the same data efficiency for coding, long-horizon agents, or frontier-scale models.
Scaling dimensions
Models, methods & benchmarks
- Models
- DeepSeek-R1-Distill-Qwen-1.5B, JustRL-DeepSeek-1.5B, DeepSeek-R1-Distill-Qwen-7B, Skywork-OR1-Math-7B, Qwen3-4B, Qwen3-1.7B-Base, Qwen3-4B-Base
- Algorithms
- On-Policy Distillation, Difficulty-based data selection, Reverse KL distillation
- Benchmarks
- AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva Math, OlympiadBench
Topics