TL;DR
Zero-Sum Calibrated Policy Optimization (ZCPO) puts reference-policy drift into group-relative update coefficients instead of adding an independent KL loss. The paper analyzes seven potential KL–reward mismatches and reports improved math and long-context performance in its tested configurations.
Why it matters
Whether KL helps depends on where it enters the update and how it interacts with reward clipping, response length, and group comparisons. ZCPO offers a concrete calibration design and diagnostic controls, while distinguishing a coefficient-level optimality result from guarantees about training performance.
Key findings
- 01
ZCPO centers rewards within each group and normalizes them by their maximum absolute centered value. It subtracts a weighted, group-centered mean per-token conditional KL, computed over the full vocabulary, and uses the resulting frozen coefficient in the base policy surrogate. Groups with fewer than two responses or identical rewards receive zero loss.
- 02
The main Qwen2.5-32B experiment uses DAPO-Math-17K in verl, with 512 prompts and 16 responses per prompt in each training batch. Dynamic sampling replenishes filtered groups, so 8,192 retained responses is not necessarily the total generation count. Each rollout batch receives 16 mini-batch updates with coefficients frozen at the sampling snapshot.
- 03
At checkpoints matched by cumulative GPU compute time, ZCPO reports AIME24/AIME25 avg@32 scores of 56.8/48.5 versus 47.3/37.1 for DAPO without KL: gains of 9.5 and 11.4 percentage points. Table 1 reports means over three random seeds; avg@32 averages correctness across attempts and is not pass@32.
- 04
DAPO with an independent KL coefficient of 0.001 scores 44.5/34.7, while a coefficient of 0.04 scores 31.2/18.9. ZCPO uses a differently constructed KL term with coefficient 4; nominal coefficients cannot be compared as equal regularization strength across objectives.
- 05
Removing KL centering lowers ZCPO to 45.6/36.7. On AIME24, retaining maximum-absolute reward normalization but disabling KL calibration yields 48.1 versus 56.8 with calibration. These controls support gains beyond changing reward normalization alone in this configuration.
- 06
The author reports approximately 0.5% additional GPU compute time relative to standard KL under the same configuration, from fp32 output-projection recomputation and full-vocabulary KL reduction. Main evaluation checkpoints are compute-matched; this is a reported implementation overhead, not a general speedup.
- 07
Additional experiments train Qwen3-4B-Thinking-2507 on the same fixed 8,000-example GoLongRL subset across methods and report gains over GRPO without KL on six long-context benchmarks. A separate GLM-4-9B-0414 math comparison reports AIME24 avg@32 of 36.2 versus 24.5 for DAPO without KL.
Evidence & limits
The unique-optimum result concerns a fixed-input, zero-sum coefficient calibration problem. It does not guarantee monotonically increasing reward, decreasing KL, or globally optimal policy training; sufficiently strong calibration can still reverse a positive reward coefficient.
Zero-sum response coefficients do not imply a zero total parameter gradient. Joint cancellation requires special assumptions about shared contexts and tokens, open clipping gates, and equal aggregation weights. Conditional KL removes current-token sampling noise given a prefix, but not variation across sampled prefixes.
The main math experiments filter identical-reward groups for both methods and therefore do not test that failure mode. The separate long-context gating ablation yields modest mixed changes, including MRCR 66.0 with gating versus 66.3 without it.
The main table reports three-seed means without dispersion in that table. The additional long-context and GLM comparisons should not be assumed to have the same seed protocol or establish broad agentic and coding generalization.
Gradient, length, and drift diagnostics support several mechanistic hypotheses but do not isolate the causal contribution of every proposed failure mode to task gains. Length normalization removes direct accumulation with length, not every residual length correlation.
The frozen-coefficient surrogate is not an unbiased reformulation of the original sequence-KL gradient. Full-vocabulary computation and fp32 precision handling are implementation requirements; the reported overhead is specific to the tested setup.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-32B, Qwen3-4B-Thinking-2507, GLM-4-9B-0414
- Algorithms
- ZCPO, DAPO, GRPO, Conditional KL calibration
- Benchmarks
- AIME24, AIME25, DocMath, LongBench-V2, Frames, MRCR, CorpusQA, LongBench-v1 QA
Topics