TL;DR
NeMo-DCR transfers compressed weight changes between training and rollout clusters while reconstructing the same parameter and buffer bits as a dense refit. Cross-region benchmarks report large synchronization speedups, with separate small-scale training experiments testing recovery from interrupted updates.
Why it matters
Disaggregated RL can waste network capacity repeatedly transferring unchanged BF16 weights. Exact delta transfer tackles that systems cost while preserving native model loading and explicit policy-version recovery, rather than approximating the updated policy.
Key findings
- 01
The system maps training-shard changes into canonical checkpoint coordinates, uses compressed XOR masks where stored bits are preserved and overwrites elsewhere, and applies updates through the serving runtime’s native loader. Overwrite retries repair partial writes; a joint commit aligns receiver policy and the baseline for the next delta.
- 02
Across six models, the first five optimizer steps change 0.6–1.2% of stored BF16 source elements per step on average under a constant 1e-6 learning-rate recipe. Latency benchmarks separately inject uniformly selected 3% and 5% changes as stress cases, rather than using complete training trajectories at every scale.
- 03
The cross-region latency testbed uses 32 GB300 GPUs on training nodes and 64 H100 GPUs on rollout nodes, with up to 5 Gbps measured cross-cluster flow per node. For the 120B checkpoint at 3% changes, relay-tree refit takes 22.6 seconds versus 750 seconds for the transport-only full-checkpoint reference, a reported 33.2× speedup.
- 04
Across 30B–1T checkpoint sizes and the two stress-case change rates, refits are reported as 12–40× faster than that reference. The 1T relay-tree point at 3% takes 150 seconds versus 87.5 minutes; this checkpoint is constructed by doubling the layers of Nemotron-3-Ultra-550B-A55B.
- 05
In a separate Qwen3-30B-A3B experiment, five runs per method train for 50 GRPO steps on swe1.jsonl from Nemotron-RL-Super-Training-Blends. Random receiver instances are killed mid-refit every five steps in the delta-transfer runs. Mean training reward is 0.41 for dense NCCL and both NeMo-DCR transports, and reported reward/KL trajectories track each other.
- 06
The training task rewards single-step tool-call argument matching, not full software-issue resolution. Dense-refit comparisons confirm bitwise equality of all parameter and buffer elements in the evaluated BF16 receiver configurations, including unchanged elements.
Evidence & limits
The headline speedups measure weight-refit latency, not end-to-end RL training throughput or downstream agent capability. The reference transfers full checkpoints through object storage across regions without a cross-cluster NCCL route.
The 1T latency result uses a synthetically expanded checkpoint and injected changes. It does not demonstrate convergence or training quality of a trillion-parameter agent; learning behavior is checked separately at 30B for 50 steps.
The measured latency window includes delta construction, compression, transfer, application, and commit, but excludes change injection, post-apply checks, initial model loading, baseline construction, and prewarming. Full operational cost includes these additional stages.
The low observed change rate depends on BF16 storage and the tested optimizer settings. Different precisions, learning rates, training phases, or conversion paths can alter sparsity and compression benefits.
Bit-exactness depends on the stated loader and version-alignment conditions. The failure tests cover injected receiver kills; they are not a proof of recovery from every infrastructure or storage failure.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-30B-A3B, Qwen3-235B-A22B, Nemotron-3-Nano-30B-A3B, Nemotron-3-Super-120B-A12B, Nemotron-3-Ultra-550B-A55B
- Algorithms
- GRPO, Delta-compressed refit, XOR/overwrite encoding
- Benchmarks
- Cross-region weight-refit latency, Nemotron-RL-Super-Training-Blends tool calls
Topics