TL;DR
RISED uses shared behavioral rubrics to select rollout groups across agent environments and to condition an on-policy self-distillation teacher. It improves mean validation pass rate on two small-model backbones, while adding judge inference, teacher forward passes, and rollout oversampling.
Why it matters
Groups whose rollouts all succeed or all fail provide no within-group reward contrast for GRPO. Rubric-conditioned self-distillation adds token-level supervision for such groups, while behavior-based selection tests a richer allocation signal than reward variance alone.
Key findings
- 01
The main setup jointly trains on ALFWorld, WebShop, and DBBench using Qwen2.5-3B-Instruct or Qwen3-4B with thinking disabled. Each of 1,000 updates selects 64 prompt groups from 128 candidates, with eight rollouts per prompt: 512 updated rollouts out of 1,024 generated candidates.
- 02
A Qwen3-32B judge tags rollouts against 59 fixed rubrics across 10 skills. Selection balances similarity to the mixed batch with redundancy among selected groups, using round-robin environment quotas. The vocabulary was constructed offline with stronger-model assistance.
- 03
Positive rubrics condition a stop-gradient teacher using the policy’s own parameters; the student receives no privileged positive context. Negative rubrics steer subsequent training rollouts when violations dominate. Validation uses original prompts without either form of injected rubric guidance.
- 04
Final-checkpoint overall pass rate is 54.2% for RISED versus 48.4% for GRPO-64 on Qwen2.5-3B, and 58.5% versus 51.6% on Qwen3-4B. These are gains of 5.8 and 6.9 percentage points; gains over the strongest baseline on each backbone are smaller, at 1.2 and 0.6 points.
- 05
Overall equally averages the three environment-level binary success rates, evaluated with four rollouts per held-out prompt. The main table reports means and standard deviations across multiple runs; RISED does not lead every environment individually.
- 06
For Qwen2.5-3B, average step time over the first 300 steps is 252.9 seconds for RISED versus 155.5 for GRPO-64 and 249.6 for GRPO-128. Rubric methods additionally use a dedicated eight-GPU judge server beyond 32 H100s for training and rollout generation, so wall-clock comparisons are not total-compute comparisons.
Evidence & limits
The main evidence covers two 3B/4B backbones and three environments. The additional AppWorld experiment has low, variable success and does not establish broad generalist-agent capability.
Reaching a baseline score in fewer optimizer steps does not by itself prove lower cost: RISED doubles candidate generation relative to GRPO-64 and adds rubric tagging and teacher inference. Asynchronous rollout wait measures blocking, not total generation work.
Gains over the strongest baselines are modest, and the reported standard deviations do not establish statistical significance. Additional ablations generally use one seed.
The fixed vocabulary and model-generated annotations introduce coverage and judgment limitations. Re-tagging agreement supports repeatability of aggregate profiles, not correctness of every behavioral label.
Rubric similarity correlates with transfer in the studied settings; that association is not a causal guarantee. Removing the distillation component also removes negative steering, so that ablation does not isolate the teacher loss alone.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-3B-Instruct, Qwen3-4B, Qwen3-32B
- Algorithms
- RISED, GRPO, On-policy self-distillation
- Benchmarks
- ALFWorld, WebShop, DBBench, AppWorld
Topics