TL;DR
RetireOPD first trains a teacher with privileged task skills, then combines on-policy distillation with RL for a student that cannot access those skills. An adaptive rule retires the teacher once transfer stalls and the student reaches sufficient task competence.
Why it matters
Dense teacher feedback can help an agent learn early and constrain it later. This study treats the end of distillation as a measurable training decision, extending the question of how to schedule OPD and reward-based optimization.
Key findings
- 01
The teacher and student begin with the same model architecture and initialization. The teacher is separately optimized with environment rewards and retrieved SkillBank information, so its advantage depends on learned use of privileged context rather than model size alone.
- 02
The student trains jointly with GRPO and OPD. Teacher supervision ends when the monitored behavior gap stops shrinking and student success reaches a target fraction of teacher success; training then continues with GRPO alone.
- 03
Across Qwen2.5-1.5B, 3B, and 7B, the method improves over GRPO on ALFWorld and WebShop. At 3B, ALFWorld success rises from 75.0% to 93.8%, an 18.8-percentage-point gain. At 1.5B, WebShop accuracy rises from 56.8% to 75.8%, a 19.0-point gain.
- 04
The student exceeds its corresponding skill-conditioned teacher in all tested model-scale and environment settings. Comparisons against continued joint training support retiring supervision after its early benefit has been absorbed.
Evidence & limits
Evidence comes from one model family at 1.5B–7B and two simulated interactive benchmarks. Transfer to open-ended software work or other agent environments is not established.
The recipe requires privileged skills and a separately optimized teacher. Student performance gains should not be interpreted as free supervision or a universal compute-matched advantage.
Retirement depends on monitored performance and behavior-gap thresholds. The experiments do not establish a universally optimal stopping rule or unbounded self-improvement.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct
- Algorithms
- RetireOPD, On-Policy Distillation, GRPO, Adaptive teacher retirement
- Benchmarks
- ALFWorld, WebShop
Topics