Compute
The hardware, memory, communication, and wall-clock budget available to generate experience and update the policy.
A practical field guide
RL Scaling
How reinforcement learning systems grow—and where they stop working.
RL Scaling studies what happens when reinforcement learning systems grow across compute, data, model capacity, rollout volume, task diversity, and feedback quality. The central question is not simply whether more resources improve a policy, but which part of the learning loop becomes the next constraint. This topic tracks empirical scaling behavior, large training systems, and the operational lessons behind durable gains.
01
The dimensions
RL does not scale along one axis.
A larger model or a larger compute budget changes only part of the learning system. Durable gains depend on how six coupled dimensions grow together.
Data
The prompts, tasks, trajectories, demonstrations, and feedback examples that define what the policy can learn.
Environments
The systems in which a policy acts—from math graders and code sandboxes to browsers, simulators, and production tools.
Rollouts
The quantity, diversity, length, and freshness of sampled trajectories used to estimate which behaviors should improve.
Rewards
The human, learned, or programmatic signals that determine which outcomes the optimization process reinforces.
Agents
The models, tools, memory, and control loops that must remain reliable as tasks become longer and more interactive.
02
The learning loop
Scale the loop, not one component.
01→
Environment
Defines the task, observations, available actions, transition rules, and outcomes.
02→
Rollouts
Turns the current policy into trajectories that expose both capability and failure.
03→
Rewards
Scores outcomes or behavior and decides which trajectories create learning pressure.
04→
Optimization
Converts sampled feedback into policy updates while managing drift and instability.
05
Evaluation
Tests whether gains are real, generalize beyond training tasks, and survive deployment constraints.
The central scaling question: when the policy stops improving, which link in this loop prevents additional resources from becoming useful capability?
03
Where scaling breaks
Find the binding constraint before adding pressure.
A plateau is an observation, not a diagnosis. The same flat reward curve can come from insufficient compute, repetitive rollouts, a saturated verifier, unstable updates, or an evaluation that cannot see the gain.
LayerFailure signalFirst diagnostic question
Compute
Workers idle, rollout throughput stalls, or memory and communication dominate run time.
Which resource is saturated, and does adding it increase useful experience per hour?
Data
Performance plateaus on familiar task families and fails to transfer to new distributions.
Is the training distribution broad enough to expose the behavior we want?
Environments
The policy exploits resets, tools, or task shortcuts that do not exist in production.
Does success in the environment require the same behavior as success in the real workflow?
Rollouts
Samples are repetitive, too short, stale, or dominated by failures with little learning value.
Are we generating enough diverse and current trajectories to distinguish better policies?
Rewards
Reward rises while human judgment, downstream tests, or safety metrics get worse.
What can the policy exploit, and which desired behaviors remain invisible to the reward?
Evaluation
Headline benchmarks improve but regressions appear in cost, latency, reliability, or transfer.
Would the claimed gain survive a different judge, seed, task set, and production constraint?
RLS
ANALYSIS
001
ANALYSIS
001
A working framework
The RL Scaling Bottleneck Map
A practical framework for locating the binding constraint across environment coverage, rollout generation, reward validity, optimization, and evaluation—before spending the next unit of compute.
Read the full analysis →- 01State the plateau
- 02List competing bottlenecks
- 03Choose a discriminating test
- 04Change one pressure at a time
04
Recent evidence · Updated Sep 1, 2026
Scale exposes trade-offs, not one-way gains.
New August 2026 preprints sharpen the scaling question. More throughput can reduce wall-clock time only when optimization is retuned; stronger pass@1 can coincide with narrower solution coverage; and long-horizon agents need credit at the level of executable decisions.
When Do Larger Batches Help Scale LLM Reinforcement Learning?
The authors separate sample efficiency from systems throughput. With batch-aware learning-rate retuning, they report up to 2.29× higher generation throughput and up to 29% lower GRPO time-to-target; without retuning, the larger batch was slower.
Read the research note →Locked at the Entrance, Open Inside
Across two controlled training setups, the authors report that RLVR improved single-sample accuracy while solution coverage fell by as much as 67%. Most of the contraction appeared before the first reasoning operation, limiting later test-time scaling.
Read the research note →Process Supervision for Agentic Policy Optimization
TASPO keeps verified outcomes as the source of update direction while using privileged supervision to redistribute credit across executable actions. The authors report a 10.6% improvement over GRPO across three agentic benchmarks.
Read the research note →All three items are new preprints, and the agentic-credit paper explicitly describes itself as work in progress. Treat the reported numbers as results from particular setups—not settled scaling laws—until they are reproduced across models, tasks, budgets, and implementations.
LessWrong discussion · 2026
Where the interpretation is being challenged.
These posts are community analysis, not peer-reviewed evidence. They are included because they make the live disagreements and failure hypotheses unusually explicit.
Some Interesting Papers on RLVR
By CarolusRenniusVitellius
A compact map of the “elicitation versus capability creation” debate, including evidence on sparse updates, solution coverage, and important counterexamples.
→LLMs are (still) mostly powered by imitative learning, not RL
By Steven Byrnes
Argues that RLVR can be pragmatically decisive while still refining strategies and heuristics largely acquired through pretraining and supervised learning.
→RLVR that rewards red teaming the training environment
By Fiora Starlight
Proposes rewarding verifiable reports of environment exploits, then using those reports to patch the training loop; the comments surface monitor and stability risks.
→Four LLM loss functions → four flavors of LLM misalignment
By Steven Byrnes
Frames automatic-verifier failures as “literal genie” behavior: a useful synthesis of reward-specific failure modes, not a substitute for causal evidence.
→05
Questions for AI teams
Turn a scaling claim into a testable decision.
Should we add more compute?
Only after identifying the saturated resource and showing that additional throughput produces diverse, high-value experience rather than more of the same signal.
How many rollouts are enough?
The useful number depends on task diversity, reward variance, policy freshness, and the decisions the samples must distinguish. Volume without coverage is not scale.
Is the verifier valid?
A verifier is useful when it measures the intended outcome, resists shortcuts, remains calibrated off-distribution, and is checked against independent evaluation.
Do benchmark gains transfer?
Treat transfer as a separate hypothesis. Test new tasks, judges, seeds, environments, budgets, and failure costs before calling a result a capability gain.
06
View the full index →Latest RL Scaling research
Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization
Jingxiao Yang
A work-in-progress study of long-horizon agent reinforcement learning that uses privileged process supervision to redistribute verified trajectory-level credit across executable actions.
The RL Scaling Bottleneck Map
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
Qiancheng Zhou
A controlled study of where reasoning diversity disappears during RLVR, distinguishing failure to initiate a solution family from failure to execute it after the first reasoning choice.
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Xingyu Shen
A method for preserving reasoning diversity during RLVR by conditioning the target model on partial trajectories produced by a smaller, weaker model.
Performance Foundations of Parallel & Distributed Reasoning Language Models
Maciej Besta
A compute-centric systematization of RL-style reasoning-model post-training, covering PPO and GRPO as well as intra-model, inter-model, synchronous, and asynchronous parallelism.
07 / Frequently asked questions
A concise RL scaling primer.
Definitions for the terms and distinctions used throughout this guide.
What is RL scaling?+
RL scaling is the study and engineering practice of increasing reinforcement learning capability across compute, models, data, environments, rollouts, rewards, and agents. It asks which resources produce durable gains and which component becomes the next constraint.
Is RL scaling the same as model scaling?+
No. Model scaling changes the capacity or pretraining of the policy. RL scaling concerns the entire learning loop, including experience generation, feedback, optimization, evaluation, and the systems that connect them.
What usually limits RL training at scale?+
Common constraints include rollout throughput, narrow task distributions, low-fidelity environments, exploitable rewards, unstable updates, and evaluations that fail to measure transfer. The binding constraint changes as the system improves.
How does RLVR change the scaling problem?+
RL with verifiable rewards can make feedback cheaper and more repeatable for tasks such as math, code, and tool use. It shifts attention toward verifier coverage, reward hacking, environment design, and the breadth of tasks that can be checked reliably.
Why do environments matter so much?+
The environment determines which experiences are available and what behavior counts as success. If it is narrow or unrealistic, more rollouts and compute can reinforce behavior that does not transfer to real use.
Need a decision, not more training volume?
Find the bottleneck in your model or agent system.
RL Scaling works with teams on frontier research, custom evaluation, and production-oriented agent systems.
Discuss a project ↗