TL;DR
WeightBridge maps trainer and rollout parameter layouts, then plans direct, balanced transfers of useful shards. On H100 deployments from 16B to 1T parameters, it sharply reduces GPU stalls during weight synchronization; the reported gains concern transfer overhead, not total RL training time.
Why it matters
Frequent policy updates require moving large checkpoints between training and rollout workers with different parallel layouts. As rollout generation becomes faster, redundant transfers and synchronization stalls can become a larger fraction of the RL iteration.
Key findings
- 01
WeightBridge automatically extracts the correspondence between trainer and rollout layouts, routes shards to useful destinations, and pipelines tensor transformation, RDMA communication, and replica reconstruction.
- 02
The study uses Miles with Megatron-LM trainers and SGLang rollout engines on H100 80GB nodes. Canonical deployments span Moonlight-16B-A3B on 2 nodes, Qwen3-30B-A3B on 4, Qwen3-235B-A22B on 16, and Kimi-K2-Instruct on 64 nodes; all use matching BF16 weights.
- 03
Average GPU Stall Time per transfer falls by factors of 25.1, 26.1, 29.4, and 16.6 versus Miles broadcast across those four deployments. Against Miles P2P, the corresponding factors are 42.0, 21.0, 9.1, and 5.4. The headline 42× is therefore one transfer-stall comparison, not an end-to-end training speedup.
- 04
In the 30B synchronization-mode comparison, fully synchronous transfer has 137.5 ms average GPU stall time versus 609.1 ms with staging on both sides. On this high-bandwidth testbed, additional CPU–GPU transfers outweigh the benefit of overlapping network communication.
Evidence & limits
The primary metric averages GPU stall time across the cluster. It differs from the full transfer interval and from complete RL iteration time; total training gains depend on how much of the iteration weight synchronization originally occupies.
The experiments use a high-bandwidth H100 cluster and specific Miles baselines. Relative gains can change with interconnects, model layouts, synchronization policies, and improvements to those baselines.
Trainer and rollout parameter precision must match. Quantization before transfer, including mismatched BF16/FP8 deployment, is proposed as future work rather than demonstrated support.
The current planner assumes a fixed worker set and trainer-initiated transfers. Elasticity and failure recovery require extensions; the small coding-agent integration experiment is not a general reliability benchmark.
Scaling dimensions
Models, methods & benchmarks
- Models
- Moonlight-16B-A3B, Qwen3-30B-A3B, Qwen3-235B-A22B, Kimi-K2-Instruct
- Algorithms
- Weight transfer planning, GRPO, GSPO
- Benchmarks
- Average GPU Stall Time, End-to-end Weight Transfer Time
Topics