When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

TL;DR

On-policy distillation can suppress a student’s stopping behavior when teacher and student favor different termination tokens. Grouping operationally equivalent EOS tokens at the probability level mitigates this mechanism, while a separate late-stage source of length inflation remains unresolved.

Why it matters

Longer rollouts increase training and inference cost, but need not mean better reasoning. Termination semantics provides a concrete diagnostic when distillation produces repetition or exhausts the generation budget.

Key findings

1. 01

   Sampled-token OPD may treat the base student’s preferred EOS token as overproduced relative to a post-trained teacher, suppressing that action without reliably transferring stopping probability to the teacher’s alternative.

2. 02

   The main pair uses Qwen3-1.7B-Base and Qwen3-4B in non-thinking mode. Llama-3.2-3B and Gemma-3-4B base/post-trained pairs provide cross-family checks; all pairs share a tokenizer and vocabulary within the pair.

3. 03

   Aligning the decoding stop set alone is insufficient. Aggregating functionally equivalent termination tokens into a shared stop action in the training objective substantially reduces mismatch-driven response length and clipping.

4. 04

   K2-Horizon-7B checkpoints reveal changing termination preferences across pretraining, midtraining, and post-training. A later increase in response length persists after termination alignment, showing that EOS mismatch is only one mechanism.

Evidence & limits

• —

  Experiments train on DAPO-Math-17K and evaluate single-turn mathematical reasoning. End-of-message, end-of-turn, and tool-handoff tokens are not necessarily interchangeable in a multi-turn agent protocol.

• —

  Semantic EOS correction does not explain or eliminate every form of length inflation. Residual late-stage dynamics remain an open question.

• —

  Prompt and grading formats affect measured accuracy and length. The paper’s Avg@16 averages sampled correctness; it should not be read as pass@16 or as evidence that every saved token improves reasoning.

Scaling dimensions

Models, methods & benchmarks

Models

Qwen3-1.7B-Base, Qwen3-4B, Llama-3.2-3B, Gemma-3-4B, K2-Horizon-7B

Algorithms

On-Policy Distillation, Semantic EOS alignment

Benchmarks

DAPO-Math-17K, AMC 2023, AIME 2024, AIME 2025

Topics

Source materials

• Full paper and experimental details ↗