TL;DR
TF-OPD combines specialists sharing one base model through activation-weighted, layer-wise least squares rather than gradient training. In an Olmo 3 7B math-and-code study, the fused model retains much of both specialists’ performance. Rollout selection and numerical precision materially affect the result; the main recipe uses teacher-generated text, not ordinary student-rollout OPD.
Why it matters
Consolidating RL specialists need not always require another gradient-training stage. This paper connects model merging to a local distillation objective and shows why repetitive rollout tokens can dominate activation statistics. It offers a useful low-cost baseline for compatible checkpoints, with much narrower evidence than a general replacement for trained distillation.
Key findings
- 01
TF-OPD minimizes each linear layer’s squared output error on specialist activations, using token-normalized second-moment matrices. The RegMean-style solve uses off-diagonal shrinkage of 0.9 and equal domain weights; embeddings, normalization gains and biases use task arithmetic. The solve is exact for the specified local quadratic objective, not for end-to-end model behavior.
- 02
The experiment combines Olmo-3.1-7B-RL-Zero-Math and Olmo-3.1-7B-RL-Zero-Code from their shared Olmo-3-1025-7B base. Each teacher generates one answer for 256 training problems: DAPO-Math-17k for mathematics and DeepCoder for programming. The final statistics retain 175 correct, finished, loop-free math answers and 92 code answers passing the problem tests.
- 03
Table 2 reports a seven-benchmark mean accuracy of 61.1% for the fused model, versus 59.8% for the math specialist and 53.6% for the code specialist. These are equal-benchmark averages, with five math and two code benchmarks, and the table compares float16 fused inference with bfloat16 specialist inference.
- 04
The final model scores 90.8% on MATH-500, 69.4% on OlympiadBench, 50.0% on AIME24, 38.3% on AIME25, 27.5% on HMMT25, 84.1% on HumanEval+ and 67.5% on MBPP+. Competition-math scores average correctness across four samples per problem, rather than counting success on any one of four attempts; the other main-table scores use one sample.
- 05
In the separate AIME25 selection ablation, all-answer statistics score 34.2%, while removing looping math answers yields 40.0%, an absolute 5.8-point increase. The float16 math specialist scores 39.2%. The final model additionally filters code answers and is a different checkpoint from this 40.0% ablation.
- 06
Appendix B reports about 10 GPU-hours for consolidating two specialists with 256 problems each on 48 GB GPUs. Generation dominates; statistics collection and linear solves add cost. No gradient updates or optimizer state are used during consolidation, but the cost excludes the original specialist RL training.
- 07
The authors identify 19 truncated math answers among 256 rollouts, all repetition loops. They detect loops using a DEFLATE compression ratio below 0.05 on the last 20,000 characters. Per-token activation statistics can overweight such long, repetitive failures even when most answers are normal.
Evidence & limits
The main procedure is on-policy with respect to each teacher’s generated text. It is not the usual student-rollout, teacher-supervised gradient OPD setup. Student-rollout variants are tested separately and do not establish equivalence between the objectives.
The method assumes a shared base and compatible architecture and tensor shapes. The empirical study uses two specialists from one 7B family; it does not establish scaling to many domains, other model sizes, or heterogeneous teachers.
Local teacher-activation matching does not guarantee that the fused network visits the same hidden states or reproduces the teacher’s final answer distribution. Shrinkage changes the unregularized least-squares objective, and non-linear-map parameters use a separate merge rule.
Numerical precision is a material confound: the AIME25 specialist scores 43.3% in bfloat16 and 39.2% in float16. Fused weights are stored in float32 and evaluated in float16. The headline seven-benchmark mean is not a uniformly precision-matched comparison.
AIME24, AIME25 and HMMT25 each contain only 30 problems. The paired 95% bootstrap interval for the 5.8-point selection improvement is −1.7 to +13.3 points, including zero. Point estimates do not establish a statistically reliable filtering gain or equality with the specialist.
Automatic filtering introduces grader dependence. Code filtering handles 241 standard-input problems; only 212 reference solutions pass the same grader. Mathematics uses rule checks plus CompassVerifier-3B for rejected answers. Selection errors can alter the retained statistics.
The reported compute is an author estimate for this implementation, not a matched end-to-end speedup over gradient-trained OPD. No trained-OPD baseline is reported in the main comparison, and “training-free” still includes generation, verification, forward passes and linear algebra.
Scaling dimensions
Models, methods & benchmarks
- Models
- Olmo-3-1025-7B, Olmo-3.1-7B-RL-Zero-Math, Olmo-3.1-7B-RL-Zero-Code
- Algorithms
- TF-OPD, RegMean, Task arithmetic
- Benchmarks
- MATH-500, OlympiadBench, AIME24, AIME25, HMMT25, HumanEval+, MBPP+
Topics