TL;DR
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
Why it matters
A stronger agent-training recipe needs evidence about failures and cost as well as favorable endpoints. This benchmark provides direct comparisons and recorded timing, making it easier to distinguish a useful task-specific intervention from a generally reliable improvement over GRPO.
Key findings
- 01
The study runs 11 methods across 12 model–task settings for 100 global steps each: 132 training runs using eight H800 GPUs per run. Tasks are ALFWorld (140 games), ScienceWorld (200 tasks), WebShop (128 goals), and Search-QA (3,125 questions across seven sources). Models are evaluated without privileged teacher context.
- 02
Final results average eight rollout seeds at temperature 0.4 and report sample standard deviations. These are repeated evaluations of trained checkpoints, not eight independent training seeds. The same fixed task sets are used for training-time validation and final evaluation.
- 03
Only PCSD and AgentOPSD exceed GRPO in the equal-weight mean across all twelve settings: 54.26 and 51.34 versus 49.68, respectively. These are aggregate score gains of 4.58 and 1.66 points across heterogeneous metrics, not universal accuracy percentage-point improvements.
- 04
No method dominates every setting. PCSD beats GRPO in six settings, trails in five, and ties in one despite having the highest aggregate score. ICSD wins seven settings but averages 44.75, with zero final success on 4B WebShop; win counts can conceal severe failures.
- 05
Of 120 method–setting comparisons against GRPO, 112 take more net training time; 58 of those also have lower mean final performance and seven tie. PCSD and AgentOPSD have mean paired net-time ratios of 1.17 and 1.19 relative to GRPO.
- 06
In the separate validation-based time-budget analysis, AgentOPSD’s ALFWorld mean across model sizes changes from 68.10% at step 100 to 37.63% at the latest checkpoint within GRPO’s net-time budget, below GRPO’s 58.10%. These validation results are distinct from the eight-seed final evaluation.
- 07
The paper records 12,124.4 H800 GPU-hours across the training loops, including 10,580.7 for training, 1,515.3 for in-loop validation, and 28.4 for checkpoint saving. Net training time includes rollout generation, teacher computation, and policy updates; equal global-step counts do not imply equal compute.
Evidence & limits
The results cover Qwen3.5 at 2B, 4B, and 9B, four text-agent benchmarks, and a 100-global-step horizon. They do not establish universal rankings for other backbones, longer training, coding agents, or every self-distillation method.
The common evaluation retains method-specific objectives and hyperparameters informed by original implementations. Skill-SD and RLSD use skill-bank adaptations, and GRPO+OPSD uses SDAR’s constant-gate implementation. Results apply to these implementations rather than every possible tuning of each method.
Rollout-seed variability measures evaluation sampling variation, not training-seed robustness or significance of differences between independently trained runs. A single observed collapse does not establish its frequency across training seeds.
The aggregate combines ALFWorld and WebShop success, Search-QA exact match, and ScienceWorld task scores with equal weight across twelve settings. It is a descriptive cross-setting summary, not a single calibrated accuracy metric.
The time-budget comparison chooses the latest available validation checkpoint within each GRPO budget. Its scores use validation trajectories, so they should not be substituted for the repeated final-evaluation means.
Training uses 128 student episodes per global step on ALFWorld, WebShop, and ScienceWorld, but 512 nominal episodes on Search-QA; some methods add teacher continuations. The reported GPU-hours describe the recorded experiment loops and are not a complete project-development or dollar-cost estimate.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B
- Algorithms
- GRPO, GRPO+OPSD, SDAR, Skill-SD, RLSD, OCSD, OVCSD, AgentOPSD, ADRS, ICSD, PCSD
- Benchmarks
- ALFWorld, ScienceWorld, WebShop, Search-QA
Topics