TL;DR
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
Why it matters
Dense token supervision is not automatically useful credit assignment. Separating error identification from teacher scoring provides a concrete way to test what the distillation signal conveys, while exposing the cost and privileged-information requirements of critique generation.
Key findings
- 01
Token-signal analysis covers Qwen3-1.7B, Olmo-3-7B-Think, and Olmo-3-32B-Think on Knights & Knaves and two DeepMath difficulty bands; Qwen3-8B is an additional scoring-teacher control. The analysis focuses on verifier-incorrect, untruncated rollouts ending in a boxed answer, with correct sibling rollouts retained for correctness prediction.
- 02
The probes measure distributional localization and whether privileged context lowers the probability of erroneous student continuations, relative to clean regions and placebo controls. Reference-only conditioning fails the joint corrective criterion in the reported settings; oracle-critique conditioning satisfies it across the main model/task combinations. This is signal evidence, not itself a task-accuracy result.
- 03
Quote-only and reason-only ablations distinguish pointing to an error from explaining it. Quotes preserve much of the localization signal, but omitting the reason weakens correction and can reverse its sign with the larger scoring teacher.
- 04
The principal training experiment uses Olmo-3-7B-Instruct on 320 Knights & Knaves puzzles for 100 steps, with 16 puzzles and eight rollouts per step. Training uses an 8,192-token cap; final evaluation uses 32,768 tokens on 500 held-out in-domain puzzles and 100 each with 9, 10, and 11 people.
- 05
OOD accuracy, averaged over the three larger puzzle sizes, is 50.9% for GRPO, 56.4% for GRPO plus gpt-5-mini critique, 56.7% for reference-conditioned self-distillation (RLCSD), and 60.6% for RLCSD plus gpt-5-mini critique. The latter is a 3.9-percentage-point gain over RLCSD; self-critique reaches 59.3%.
- 06
The main training critique term is applied to wrong rollouts in mixed-outcome groups, on the quoted sentence and 50 following tokens. A separate one-seed, four-puzzle cold-start pilot uses a different all-wrong-group intervention; its solved training rollouts should not be confused with held-out generalization.
- 07
For the main training runs, the in-engine self-critique pass represents 6% of step time and averages 175 completion tokens per critique. API-critic runs spend 21–22% of step time waiting for critiques, including about 3,000 completion/reasoning tokens per critique. API waiting time is not GPU compute time.
Evidence & limits
Every critique in the analysis, training, and inference experiments sees a reference solution. The results establish a use of privileged feedback, not unaided self-correction or a deployment method for problems without references.
Training evidence is limited to 320 puzzles and 100 steps, plus a separate four-puzzle pilot. The main training table does not establish multi-seed robustness or full-scale RL gains; improvements should not be generalized to coding or long-horizon agents.
The main training cost measurements critique only 16–18 of 128 rollouts per step in mixed-outcome groups. A 6% share of step time is not a measured 6% overhead against a matched baseline, and the cost cannot be extrapolated directly to all-wrong batches.
Most error-span labels are model-generated and accepted by two model judges, with placebo checks and an additional human-annotated ProcessBench analysis. Selection of wrong, complete rollouts and dependence on error annotations constrain the interpretation.
The test-time experiment contains 55 wrong rollouts from 39 DeepMath problems. On 19 problems unsolved in eight initial samples, sample accuracy rises from 4.3% to 9.3%, but the intervention changes both cut location and critique content and uses the reference; this is neither pass@8 nor a clean content-only ablation.
The self-critique training implementation temporarily loads frozen base weights for critique generation. Its result should not be described as proof that an evolving student autonomously improves its own critic, and the signal findings do not imply that all reference-conditioned distillation is ineffective.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-1.7B, Qwen3-8B, Olmo-3-7B-Think, Olmo-3-32B-Think, Olmo-3-7B-Instruct
- Algorithms
- GRPO, RLCSD, Critique-conditioned self-distillation
- Benchmarks
- Knights & Knaves, DeepMath, ProcessBench
Topics