The RL Scaling Bottleneck Map
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Research index
What is actually
moving the frontier?
Source-linked research notes with original summaries and a clear view of why each development matters for scaling intelligent systems.
18 items
When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Performance Foundations of Parallel & Distributed Reasoning Language Models
Maciej Besta
A compute-centric systematization of RL-style reasoning-model post-training, covering PPO and GRPO as well as intra-model, inter-model, synchronous, and asynchronous parallelism.
GLM-5.3: Frontier Coding with Emergent Cyber Capabilities
Z.ai
An official GLM-5.3 release note describing a post-training-only update on the same base model as GLM-5.2, with expanded executable environments for complex coding, security, and long-horizon professional workflows.
Kimi K3: Open Frontier Intelligence
Moonshot AI
A technical report on Kimi K3, a 2.8T-parameter sparse multimodal model with 104B active parameters and a one-million-token context window, post-trained with reinforcement learning across general, agentic, coding, and reasoning domains.
GR2 Technical Report
Yufei Li
A production-oriented technical report on a generative reasoning re-ranker trained with semantic-ID mid-training, teacher-trace distillation, on-policy distillation, and reinforcement learning from verifiable ranking rewards.
Sakana Fugu Technical Report
Sakana AI
A technical report on Fugu and Fugu-Ultra, language-model orchestrators trained to construct query-adaptive scaffolds for teams of heterogeneous LLM agents.
GLM-5.2: Built for Long-Horizon Tasks
Z.ai
An official release of the 750B-A40B GLM-5.2 model, combining a one-million-token context, IndexShare sparse-attention reuse, and larger-scale agentic reinforcement learning for long-horizon tasks.
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Ang Li
A model-family report on Ling-2.6 and Ring-2.6, combining architectural migration, long-context efficiency work, token-efficient reasoning objectives, and asynchronous agent reinforcement learning at trillion-parameter scale.
MiniMax-M3: Native Multimodal Intelligence at 1M Context
MiniMax
An official release of MiniMax-M3, a natively multimodal sparse model with about 428B total and 23B active parameters, a one-million-token context window, and agent-oriented coding and cowork capabilities.
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
MiniMax
A technical report on the MiniMax-M2 family, pairing a 229.9B-parameter sparse MoE with agent-generated, verifiable trajectories and Forge, a scalable reinforcement-learning system for long-horizon agents.
Composer 2 Technical Report
Cursor
A technical report on Composer 2, a specialized coding model trained through continued pretraining followed by large-scale reinforcement learning on long-horizon software-engineering tasks.
GLM-5: from Vibe Coding to Agentic Engineering
Z.ai
A GLM-5 technical report centered on agentic engineering, combining a more efficient long-context architecture with asynchronous reinforcement-learning infrastructure and agent RL for complex, long-horizon software tasks.
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
DAPO Team
DAPO presents an open reinforcement-learning system and a collection of training interventions aimed at improving large-scale reasoning optimization.
Kimi k1.5: Scaling Reinforcement Learning with LLMs
Kimi Team
A technical report on scaling multimodal language-model reinforcement learning with long-context training, policy optimization, and data-selection strategies.
Scaling Laws for Reward Model Overoptimization
OpenAI
An empirical study of how optimizing against a learned reward model can eventually reduce performance under a higher-quality reference signal.
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI
The OpenAI Five report documents a distributed, continually trained self-play system for a long-horizon, imperfect-information team game.
Proximal Policy Optimization Algorithms
OpenAI
PPO introduces clipped and adaptive objectives intended to make policy-gradient optimization simpler to implement while limiting excessively large policy updates.