TL;DR
VenusRL separates training, rollout generation, and tool environments, prioritizing trajectory groups that can unblock the next training batch. It combines priority-aware scheduling and KV-cache management with denser sandbox placement, reporting workload-dependent throughput gains on long-horizon software-agent training.
Why it matters
Keeping generation GPUs busy is not sufficient if a trainer is waiting for complete rollout groups or tool sandboxes cannot sustain concurrency. VenusRL treats group completion and environment memory capacity as connected bottlenecks in agentic RL scaling.
Key findings
- 01
The scheduler predicts remaining generation length to prioritize critical groups, protects reusable KV cache, and migrates requests across workers. Trajectories reaching their last admissible weight version are promoted to prevent further priority-based delay.
- 02
Evaluation uses instruction-tuned Qwen3-4B and Qwen3-32B on OpenSWE software-agent workloads, with eight GRPO samples per prompt, a 128K output-token limit, and at most 300 turns. Four servers provide 32 Hopper GPUs, split equally between generation and training; the compared systems use SGLang 0.5.17.
- 03
Across tested batch sizes and asynchronous settings, the authors report 1.07–3.26× end-to-end training throughput relative to Slime, 1.06–4.24× relative to RollFlash, and 1.05–2.67× relative to ThunderAgent. These ranges compare particular workload configurations, not model accuracy.
- 04
Under strict on-policy operation, reported gains are smaller: 1.07–1.15× for Qwen3-4B and 1.05–1.85× for Qwen3-32B across the compared settings. Larger asynchronous batches offer more opportunity to reduce group-completion delays and cache contention.
- 05
On a 400 GB host with sandboxes cloned from one template, the reported capacity rises from 100 under native E2B admission to 743 with footprint-aware placement and 905 with page sharing. The final density is 9.05× the baseline, not evidence of an equivalent reduction in total RL training cost.
- 06
At batch size 32 and async factor 1.5, the displayed training-reward curves for Slime, RollFlash, and VenusRL converge to similar rewards over comparable steps. Sandbox migration takes 2.70 seconds versus 3.68 for the native E2B recovery path in the reported experiment.
Evidence & limits
The study focuses on long-horizon OpenSWE workloads, two Qwen3 sizes, and one cluster configuration. Maximum throughput gains should not be extrapolated to short rollouts, other environments, or arbitrary hardware.
The representative convergence plot measures training rewards, not held-out software-task success. Similar displayed curves do not establish broad equivalence in downstream quality or robustness across training seeds.
The abstract’s up-to-89% environment-cost claim concerns environment provisioning. The concrete density experiment uses same-template sandboxes on a 400 GB node; it does not demonstrate an 89% reduction in total training spend.
The remaining-length predictor is trained on trajectories matching the rollout temperature. Workload or policy shifts can require recalibration; the reported rank correlation does not guarantee reliable ordering outside that distribution.
Footprint-based admission needs workload-specific memory headroom, and page-sharing benefits depend on shared state. Reported lifecycle latencies also include cases with higher pause/resume overhead, so the environment manager is not uniformly free of overhead.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3-4B, Qwen3-32B
- Algorithms
- GRPO, Priority-aware rollout scheduling, Copy-on-write sandbox page sharing
- Benchmarks
- OpenSWE
Topics