TL;DR
CLOOPD reuses each teacher-scored batch for multiple student updates, separating the cost of acquiring supervision from the compute spent learning it. In one math-distillation setup, three actor passes nearly match a longer TOP-D run with fewer teacher tokens and allocated GPU-hours, while higher endpoint quality costs more compute.
Why it matters
Teacher-token efficiency and total training efficiency are different objectives. CLOOPD makes this tradeoff explicit: extra student updates can amortize expensive teacher inference, but the best final score does not imply the cheapest training run.
Key findings
- 01
CLOOPD selects an adaptive alpha waypoint within a KL envelope, freezes the scored batch and advantages, and re-forwards the student after each actor pass. Two-pass, three-pass, selective TPMR, and budget-matched random policies reuse supervision without additional teacher scoring.
- 02
The main study trains Qwen3-1.7B-Base from Qwen3-30B-A3B-Instruct-2507 on DAPO-Math-17k. Six 300-step runs use seed 42 on eight H20 GPUs: four for actor/rollout work and four for the teacher.
- 03
At step 300, Fixed3 reaches 19.36 macro accuracy versus 17.78 for Fixed2 and 15.41 for TOP-D, at roughly 292M teacher-scored tokens each. Allocated GPU-hours are 288.1, 216.7, and 131.5 respectively. The cleanest added-pass comparison is Fixed3 versus Fixed2; comparison with TOP-D changes other framework components too.
- 04
At step 100, Fixed3 scores 15.346 versus TOP-D’s 15.406 at step 300, a 0.060-point shortfall. It uses 95.9M versus 292.5M teacher tokens (67.2% fewer) and 94.7 versus 131.5 allocated H20 GPU-hours (28.0% fewer), with similar actor-token volume.
- 05
TPMR reaches 17.28 at step 300, below the random control’s 18.47. The results do not establish that the proposed selective allocator beats random allocation.
- 06
Macro accuracy equally weights AIME 2024/2025/2026, AMC 2023, and MATH-500. Evaluation averages generation-level exact-verifier correctness over 32 samples per AIME/AMC problem and four per MATH-500 problem; it is not pass@k.
Evidence & limits
The main evidence covers one student–teacher pair, one math dataset, and one training seed. Multiple generations and checkpoints do not establish training-seed robustness.
The near-quality milestone is 0.060 points below the comparator on a finite checkpoint grid; statistical equivalence is not established. The higher step-300 Fixed3 score costs more than twice TOP-D’s allocated GPU-hours.
Teacher tokens count scored response tokens; actor tokens count response tokens across update passes. Allocated GPU-hours equal wall time times eight occupied GPUs. These are different budgets, and token counts are not hardware-independent FLOP estimates.
Earlier 50–150-step A100 mechanism studies are a separate evidence tier from the H20 main runs and do not support direct cross-hardware efficiency comparisons.
Controller calibration, benchmark weighting, and relative teacher/actor resource costs can change the preferred operating point. Generalization beyond this math-distillation implementation remains untested.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-1.7B-Base, Qwen3-30B-A3B-Instruct-2507
- Algorithms
- On-policy distillation, CLOOPD, TOP-D
- Benchmarks
- AIME 2024, AIME 2025, AIME 2026, AMC 2023, MATH-500
Topics