The RL Scaling Bottleneck Map
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Research index
What is actually
moving the frontier?
Source-linked research notes with original summaries and a clear view of why each development matters for scaling intelligent systems.
21 items
When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Xingyu Shen
A method for preserving reasoning diversity during RLVR by conditioning the target model on partial trajectories produced by a smaller, weaker model.
LLMs are (still) mostly powered by imitative learning, not RL
Steven Byrnes
A community argument that RLVR is operationally important but that most language-model knowledge and available reasoning strategies still originate in pretraining and supervised imitation.
GR2 Technical Report
Yufei Li
A production-oriented technical report on a generative reasoning re-ranker trained with semantic-ID mid-training, teacher-trace distillation, on-policy distillation, and reinforcement learning from verifiable ranking rewards.
Some Interesting Papers on RLVR
CarolusRenniusVitellius
A concise reading map of weight-level and behavioral evidence on whether RLVR mainly reweights existing capabilities or creates new reasoning mechanisms.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI
DeepSeek-R1 documents a reasoning-model training pipeline centered on reinforcement learning, including a zero-style RL stage and a staged route that combines supervised data with RL.
Kimi k1.5: Scaling Reinforcement Learning with LLMs
Kimi Team
A technical report on scaling multimodal language-model reinforcement learning with long-context training, policy optimization, and data-selection strategies.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
DeepSeek-AI
DeepSeekMath combines a math-focused data pipeline with supervised fine-tuning and reinforcement learning, and introduces Group Relative Policy Optimization.
Self-Rewarding Language Models
Meta FAIR
This work studies language models that generate candidate responses and also provide the preference signal used to improve subsequent iterations.
Reinforced Self-Training (ReST) for Language Modeling
Google DeepMind
ReST alternates offline sample generation with policy improvement, allowing generated data to be reused rather than coupling every update to fresh online rollouts.
Let's Verify Step by Step
OpenAI
A comparison of outcome supervision and process supervision for mathematical reasoning, centered on whether feedback should evaluate only the final answer or intermediate steps as well.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Stanford University
DPO reformulates preference optimization as a direct classification-style objective, avoiding an explicit reward-model training stage and online policy optimization.
Voyager: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang
Voyager combines an automatic curriculum, an executable skill library, and iterative prompting to build an open-ended Minecraft agent without model parameter updates.
Mastering Diverse Domains through World Models
Google DeepMind
DreamerV3 presents a world-model-based reinforcement-learning algorithm designed to work across varied domains with a shared configuration.
Constitutional AI: Harmlessness from AI Feedback
Anthropic
Constitutional AI uses written principles and model-generated critiques and preferences to reduce direct dependence on human harmlessness labels during alignment training.
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
NVIDIA
MineDojo combines a Minecraft simulation suite, a broad task set, and internet-scale multimodal knowledge for research on generalist embodied agents.
Training Language Models to Follow Instructions with Human Feedback
OpenAI
The InstructGPT work combines demonstrations, preference comparisons, reward modeling, and PPO to align language-model behavior with user instructions.
Emergent Complexity and Zero-Shot Transfer via Unsupervised Environment Design
Michael Dennis
This work frames task-distribution design as an optimization problem and introduces PAIRED, where an environment generator uses regret to create solvable but challenging curricula.
Learning to Summarize from Human Feedback
OpenAI
A large-scale study of training summarization models from human comparisons using reward modeling and reinforcement learning.
Deep Reinforcement Learning from Human Preferences
OpenAI
A foundational study of learning a reward model from pairwise human preferences and using that learned reward to train an RL agent.