TL;DR
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
Why it matters
Distillation quality depends on which student histories receive teacher feedback, as well as on the teacher signal itself. BranchOPD makes the coverage-versus-generation tradeoff concrete, while its PBR controls show why protocol fixes must be separated from the contribution of branching.
Key findings
- 01
The default samples one main trajectory, selects up to three forks, and samples two continuations per fork: at most seven trajectories per input. Fork scoring mixes rank-normalized student entropy and teacher–student disagreement. Shared prefixes are counted once in the token-mean loss, and unsuccessful branches are retained.
- 02
Mathematics distills Qwen3-8B into Qwen3-4B-Base using 17,398 DAPO-Math examples over two epochs. Agent experiments use Qwen3-4B students and Qwen3-30B-A3B teachers; the WebShop and Search-QA teacher checkpoints have additional GRPO training. BranchOPD training disables task rewards and uses teacher supervision plus protocol constraints.
- 03
Across MATH-500, AIME24, and AIME25, the macro-average avg@16 rises from 25.82 for Vanilla OPD to 44.37 for BranchOPD, a gain of 18.55 percentage points. With PBR enabled for the baselines, Vanilla OPD reaches 38.94 and seven-rollout OPD reaches 39.09, narrowing the corresponding gains to 5.43 and 5.28 points.
- 04
BranchOPD reports ALFWorld ID/OOD success of 57.86/55.22, averaging 56.54, versus 49.27 for Vanilla OPD with PBR. WebShop success is 33.59 versus 30.47, while Search-QA is 49.05 versus 48.58. Agent evaluation uses one trajectory per task, with no evaluation-time branching.
- 05
Relative to seven independent trajectories, measured student-generated tokens decrease from 19,691 to 15,924 on ALFWorld (19.1%) and from 7,014 to 4,824 on WebShop (31.2%). These measurements exclude teacher scoring, backpropagation, and environment replay; the branching runs still use more generation than single-trajectory OPD.
- 06
With PBR held fixed, random fork selection scores 40.11 on mathematics versus 44.37 for entropy-plus-disagreement selection. Increasing continuations per fork from two to three does not improve the tested results, so more branches are not uniformly better.
- 07
PBR encourages termination after a complete mathematical answer and suppresses invalid leading closing-thought tags in agent turns. PBR alone raises Vanilla OPD mathematics accuracy from 25.82 to 38.94, showing that a substantial part of the headline improvement comes from protocol repair.
Evidence & limits
Generation-token savings are relative to seven independent rollouts, not to Vanilla OPD or total training cost. Mathematics and Search-QA savings are estimates from fork positions, unlike the measured ALFWorld and WebShop token counts.
Branching in agent tasks requires restoring the environment state. A separate three-slot ALFWorld experiment uses 2.55 times Vanilla OPD generation but 3.02 times its environment steps, illustrating replay overhead that token counts omit.
The main results combine branching with PBR. Comparisons against baselines without PBR do not isolate the effect of branching; the PBR-matched controls are the more informative comparison for that purpose.
Mathematics avg@16 averages answer correctness over 16 samples and is not pass@16. Agent results use one sampled trajectory per task; WebShop has 128 held-out tasks, and Search-QA training-set size is unreported in the configuration table. The main tables do not report multi-training-seed uncertainty.
Conditional unbiasedness applies to suffix statistics at selected forks under correct context restoration and student-policy sampling. It does not establish that adaptive fork selection gives an unbiased estimate of the original full-trajectory objective or guarantees better task performance.
The student and teacher share an explicitly aligned EOS configuration in mathematics, and the agent protocol repair targets specific Qwen boundary tokens. Benefits may depend on these protocol and environment choices; broader coding and production-agent generalization remains unestablished.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-4B-Base, Qwen3-8B, Qwen3-4B, Qwen3-30B-A3B
- Algorithms
- BranchOPD, On-policy distillation, Protocol-boundary repair
- Benchmarks
- MATH-500, AIME24, AIME25, ALFWorld, WebShop, Search-QA
Topics