TL;DR
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
Why it matters
Action semantics can provide a teacher interface across model families and tokenizers. The paper makes that interface operational with same-state candidate groups, but also shows why teacher-guided state selection, semantic-matcher errors, and unequal collection costs must be considered when comparing distillation methods.
Key findings
- 01
At each visited state, the default samples four student responses and three teacher responses. Tool names and canonicalized arguments are checked programmatically, with an LLM resolving remaining semantic differences; message matching compares intent and factual content. Invalid actions receive a penalty, and additional teacher matches add a frequency bonus.
- 02
One highest-reward student action advances the environment, while all candidates in retained, non-equal-reward groups enter policy optimization. This gives teacher-guided state visitation rather than ordinary single-sample student rollouts. Group-centered, batch-whitened advantages drive a clipped token-level surrogate over the student’s own reasoning and action tokens.
- 03
Transfer training uses 6,697 curated tasks: 5,942 AWM tasks from 794 environments and 755 EnvScaler tasks from 49 environments. Four student–teacher configurations pair Qwen3-4B/8B with Qwen3.7-Flash or DeepSeek-V4-Flash, each for 100 collection steps and 64 task starts per step.
- 04
All four configurations improve the equally weighted four-benchmark summary by 3.3–6.9 points over their base models, computed before display rounding. Evaluation covers BFCL, local-only ToolSandbox, AppWorld, and Tau tasks excluded from transfer training. This summary mixes accuracy, similarity, task completion, and single-trial success rather than measuring one universal accuracy.
- 05
In a separate Tau comparison using Qwen3-4B and a Qwen3-32B teacher, held-out mean single-trial success is 40.0% for ActionOPD, 39.6% for sampled-token OPD, and 37.5% for offline KD. Four-trial all-success reliability is 14.2%, 17.5%, and 19.2%, respectively: ActionOPD does not dominate both metrics.
- 06
Self-ActionOPD uses shared current teacher/student weights but gives the teacher private task context and reference actions during collection. At step 50, held-out mean single-trial success increases from 29.2% to 34.0%, and four-trial all-success reliability from 7.5% to 10.0%. Student evaluation uses public context only.
Evidence & limits
Tau pass^4 means success in all four scheduled trials, not success on at least one attempt. All Tau evaluations disable natural-language assertions, so a successful task score does not certify every utterance.
Each configuration is trained once. Repeated interaction trials measure evaluation variability, not robustness across independent training runs; a 0.4-point advantage over OPD is not established as statistically reliable.
Equal task starts and collection steps do not equalize compute. KD and OPD collect four full trajectories per task start, whereas ActionOPD samples four candidates per state and executes one; token counts, teacher queries, matching costs, and filtering differ.
Semantic agreement can reinforce a teacher’s wrong or suboptimal action. The LLM matcher can produce false positives and negatives, and fixed sample groups may miss useful matches when plausible actions are dispersed.
The frequency bonus changes the ideal teacher-action-likelihood objective. Group normalization, masking, clipping, and teacher-guided state visitation mean the practical updates are not unbiased gradients of the original fixed-state objective.
Self-ActionOPD is not free of external assistance: private user summaries are prepared offline with DeepSeek-V4-Flash and reviewed, and separate models handle semantic matching and user simulation. Two student sizes and two teachers do not establish a general scaling law.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-4B, Qwen3-8B, Qwen3-32B, Qwen3.7-Flash, DeepSeek-V4-Flash
- Algorithms
- ActionOPD, Self-ActionOPD, On-policy distillation, Semantic action matching
- Benchmarks
- BFCL Multi-turn, ToolSandbox, AppWorld, Tau2-bench
Topics