Compute
The hardware, memory, communication, and wall-clock budget available to generate experience and update the policy.
A practical field guide
RL Scaling
How reinforcement learning systems grow—and where they stop working.
RL Scaling studies what happens when reinforcement learning systems grow across compute, data, model capacity, rollout volume, task diversity, and feedback quality. The central question is not simply whether more resources improve a policy, but which part of the learning loop becomes the next constraint. This topic tracks empirical scaling behavior, large training systems, and the operational lessons behind durable gains.
01
The dimensions
RL does not scale along one axis.
A larger model or a larger compute budget changes only part of the learning system. Durable gains depend on how six coupled dimensions grow together.
Data
The prompts, tasks, trajectories, demonstrations, and feedback examples that define what the policy can learn.
Environments
The systems in which a policy acts—from math graders and code sandboxes to browsers, simulators, and production tools.
Rollouts
The quantity, diversity, length, and freshness of sampled trajectories used to estimate which behaviors should improve.
Rewards
The human, learned, or programmatic signals that determine which outcomes the optimization process reinforces.
Agents
The models, tools, memory, and control loops that must remain reliable as tasks become longer and more interactive.
02
The learning loop
Scale the loop, not one component.
01→
Environment
Defines the task, observations, available actions, transition rules, and outcomes.
02→
Rollouts
Turns the current policy into trajectories that expose both capability and failure.
03→
Rewards
Scores outcomes or behavior and decides which trajectories create learning pressure.
04→
Optimization
Converts sampled feedback into policy updates while managing drift and instability.
05
Evaluation
Tests whether gains are real, generalize beyond training tasks, and survive deployment constraints.
The central scaling question: when the policy stops improving, which link in this loop prevents additional resources from becoming useful capability?
03
Where scaling breaks
Find the binding constraint before adding pressure.
A plateau is an observation, not a diagnosis. The same flat reward curve can come from insufficient compute, repetitive rollouts, a saturated verifier, unstable updates, or an evaluation that cannot see the gain.
LayerFailure signalFirst diagnostic question
Compute
Workers idle, rollout throughput stalls, or memory and communication dominate run time.
Which resource is saturated, and does adding it increase useful experience per hour?
Data
Performance plateaus on familiar task families and fails to transfer to new distributions.
Is the training distribution broad enough to expose the behavior we want?
Environments
The policy exploits resets, tools, or task shortcuts that do not exist in production.
Does success in the environment require the same behavior as success in the real workflow?
Rollouts
Samples are repetitive, too short, stale, or dominated by failures with little learning value.
Are we generating enough diverse and current trajectories to distinguish better policies?
Rewards
Reward rises while human judgment, downstream tests, or safety metrics get worse.
What can the policy exploit, and which desired behaviors remain invisible to the reward?
Evaluation
Headline benchmarks improve but regressions appear in cost, latency, reliability, or transfer.
Would the claimed gain survive a different judge, seed, task set, and production constraint?
RLS
ANALYSIS
001
ANALYSIS
001
A working framework
The RL Scaling Bottleneck Map
A practical framework for locating the binding constraint across environment coverage, rollout generation, reward validity, optimization, and evaluation—before spending the next unit of compute.
Read the full analysis →- 01State the plateau
- 02List competing bottlenecks
- 03Choose a discriminating test
- 04Change one pressure at a time
04
Recent evidence · Updated Sep 1, 2026
Scale exposes trade-offs, not one-way gains.
New August 2026 preprints sharpen the scaling question. More throughput can reduce wall-clock time only when optimization is retuned; stronger pass@1 can coincide with narrower solution coverage; and long-horizon agents need credit at the level of executable decisions.
When Do Larger Batches Help Scale LLM Reinforcement Learning?
The authors separate sample efficiency from systems throughput. With batch-aware learning-rate retuning, they report up to 2.29× higher generation throughput and up to 29% lower GRPO time-to-target; without retuning, the larger batch was slower.
Read the research note →Locked at the Entrance, Open Inside
Across two controlled training setups, the authors report that RLVR improved single-sample accuracy while solution coverage fell by as much as 67%. Most of the contraction appeared before the first reasoning operation, limiting later test-time scaling.
Read the research note →Process Supervision for Agentic Policy Optimization
TASPO keeps verified outcomes as the source of update direction while using privileged supervision to redistribute credit across executable actions. The authors report a 10.6% improvement over GRPO across three agentic benchmarks.
Read the research note →All three items are new preprints, and the agentic-credit paper explicitly describes itself as work in progress. Treat the reported numbers as results from particular setups—not settled scaling laws—until they are reproduced across models, tasks, budgets, and implementations.
LessWrong discussion · 2026
Where the interpretation is being challenged.
These posts are community analysis, not peer-reviewed evidence. They are included because they make the live disagreements and failure hypotheses unusually explicit.
Some Interesting Papers on RLVR
By CarolusRenniusVitellius
A compact map of the “elicitation versus capability creation” debate, including evidence on sparse updates, solution coverage, and important counterexamples.
→LLMs are (still) mostly powered by imitative learning, not RL
By Steven Byrnes
Argues that RLVR can be pragmatically decisive while still refining strategies and heuristics largely acquired through pretraining and supervised learning.
→RLVR that rewards red teaming the training environment
By Fiora Starlight
Proposes rewarding verifiable reports of environment exploits, then using those reports to patch the training loop; the comments surface monitor and stability risks.
→Four LLM loss functions → four flavors of LLM misalignment
By Steven Byrnes
Frames automatic-verifier failures as “literal genie” behavior: a useful synthesis of reward-specific failure modes, not a substitute for causal evidence.
→05
Questions for AI teams
Turn a scaling claim into a testable decision.
Should we add more compute?
Only after identifying the saturated resource and showing that additional throughput produces diverse, high-value experience rather than more of the same signal.
How many rollouts are enough?
The useful number depends on task diversity, reward variance, policy freshness, and the decisions the samples must distinguish. Volume without coverage is not scale.
Is the verifier valid?
A verifier is useful when it measures the intended outcome, resists shortcuts, remains calibrated off-distribution, and is checked against independent evaluation.
Do benchmark gains transfer?
Treat transfer as a separate hypothesis. Test new tasks, judges, seeds, environments, budgets, and failure costs before calling a result a capability gain.
06
View the full index →Latest RL Scaling research
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Zhejiang University
KnowGPO groups search-agent steps by accumulated task-relevant knowledge and rewards newly acquired facts. Controlled Qwen3-8B comparisons show gains over GRPO on four evaluation splits, but the method requires reference answers, successful training trajectories, and an external LLM verifier. Several narrative claims in the first version disagree with its tables.
ActionOPD: Distilling Decisions, Not Tokens
Tsinghua University
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
Zhejiang University
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning
Fin.ai
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO
Dalian Maritime University
A GSM8K study separates current-policy sample selection from progression between difficulty bands in GRPO. With Qwen2.5-1.5B-Instruct and LoRA, a staged curriculum reaching moderately harder problems improves test accuracy over persistent medium-difficulty selection, while a further difficulty increase reverses the gain across three seeds.
Branching On-Policy Distillation
Tsinghua University
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
07 / Frequently asked questions
A concise RL scaling primer.
Definitions for the terms and distinctions used throughout this guide.
What is RL scaling?+
RL scaling is the study and engineering practice of increasing reinforcement learning capability across compute, models, data, environments, rollouts, rewards, and agents. It asks which resources produce durable gains and which component becomes the next constraint.
Is RL scaling the same as model scaling?+
No. Model scaling changes the capacity or pretraining of the policy. RL scaling concerns the entire learning loop, including experience generation, feedback, optimization, evaluation, and the systems that connect them.
What usually limits RL training at scale?+
Common constraints include rollout throughput, narrow task distributions, low-fidelity environments, exploitable rewards, unstable updates, and evaluations that fail to measure transfer. The binding constraint changes as the system improves.
How does RLVR change the scaling problem?+
RL with verifiable rewards can make feedback cheaper and more repeatable for tasks such as math, code, and tool use. It shifts attention toward verifier coverage, reward hacking, environment design, and the breadth of tasks that can be checked reliably.
Why do environments matter so much?+
The environment determines which experiences are available and what behavior counts as success. If it is narrow or unrealistic, more rollouts and compute can reinforce behavior that does not transfer to real use.