The RL Scaling Bottleneck Map
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Topic
Scaling Laws
Scaling laws describe empirical relationships between resources and performance. In reinforcement learning, useful laws must account for interaction data, reward quality, environment diversity, optimization pressure, and failure under proxy objectives—not only parameter count and training compute.
Compute-performance curves
Data scaling
Reward overoptimization
Emergent capability
Latest research
When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
Qiancheng Zhou
A controlled study of where reasoning diversity disappears during RLVR, distinguishing failure to initiate a solution family from failure to execute it after the first reasoning choice.
Performance Foundations of Parallel & Distributed Reasoning Language Models
Maciej Besta
A compute-centric systematization of RL-style reasoning-model post-training, covering PPO and GRPO as well as intra-model, inter-model, synchronous, and asynchronous parallelism.
Kimi K3: Open Frontier Intelligence
Moonshot AI
A technical report on Kimi K3, a 2.8T-parameter sparse multimodal model with 104B active parameters and a one-million-token context window, post-trained with reinforcement learning across general, agentic, coding, and reasoning domains.
LLMs are (still) mostly powered by imitative learning, not RL
Steven Byrnes
A community argument that RLVR is operationally important but that most language-model knowledge and available reasoning strategies still originate in pretraining and supervised imitation.
GLM-5.2: Built for Long-Horizon Tasks
Z.ai
An official release of the 750B-A40B GLM-5.2 model, combining a one-million-token context, IndexShare sparse-attention reuse, and larger-scale agentic reinforcement learning for long-horizon tasks.
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Ang Li
A model-family report on Ling-2.6 and Ring-2.6, combining architectural migration, long-context efficiency work, token-efficient reasoning objectives, and asynchronous agent reinforcement learning at trillion-parameter scale.
MiniMax-M3: Native Multimodal Intelligence at 1M Context
MiniMax
An official release of MiniMax-M3, a natively multimodal sparse model with about 428B total and 23B active parameters, a one-million-token context window, and agent-oriented coding and cowork capabilities.
Some Interesting Papers on RLVR
CarolusRenniusVitellius
A concise reading map of weight-level and behavioral evidence on whether RLVR mainly reweights existing capabilities or creates new reasoning mechanisms.
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
MiniMax
A technical report on the MiniMax-M2 family, pairing a 229.9B-parameter sparse MoE with agent-generated, verifiable trajectories and Forge, a scalable reinforcement-learning system for long-horizon agents.
Mastering Diverse Domains through World Models
Google DeepMind
DreamerV3 presents a world-model-based reinforcement-learning algorithm designed to work across varied domains with a shared configuration.
Scaling Laws for Reward Model Overoptimization
OpenAI
An empirical study of how optimizing against a learned reward model can eventually reduce performance under a higher-quality reference signal.
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI
The OpenAI Five report documents a distributed, continually trained self-play system for a long-horizon, imperfect-information team game.