Independent research notes
Papers, model reports, and analysis on how reinforcement learning systems improve—and where they reach their limits. Original sources, key findings, and the questions still open.
Zhejiang University
KnowGPO groups search-agent steps by accumulated task-relevant knowledge and rewards newly acquired facts. Controlled Qwen3-8B comparisons show gains over GRPO on four evaluation splits, but the method requires reference answers, successful training trajectories, and an external LLM verifier. Several narrative claims in the first version disagree with its tables.
Tsinghua University
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
Zhejiang University
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
Fin.ai
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
Dalian Maritime University
A GSM8K study separates current-policy sample selection from progression between difficulty bands in GRPO. With Qwen2.5-1.5B-Instruct and LoRA, a staged curriculum reaching moderately harder problems improves test accuracy over persistent medium-difficulty selection, while a further difficulty increase reverses the gain across three seeds.
Tsinghua University
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
Z.ai
An official GLM-5.3 release note describing a post-training-only update on the same base model as GLM-5.2, with expanded executable environments for complex coding, security, and long-horizon professional workflows.
Moonshot AI
A technical report on Kimi K3, a 2.8T-parameter sparse multimodal model with 104B active parameters and a one-million-token context window, post-trained with reinforcement learning across general, agentic, coding, and reasoning domains.
Z.ai
An official release of the 750B-A40B GLM-5.2 model, combining a one-million-token context, IndexShare sparse-attention reuse, and larger-scale agentic reinforcement learning for long-horizon tasks.
MiniMax
An official release of MiniMax-M3, a natively multimodal sparse model with about 428B total and 23B active parameters, a one-million-token context window, and agent-oriented coding and cowork capabilities.
Z.ai
An official GLM-5.1 model update focused on keeping an agent productive across longer coding and engineering runs through repeated execution, inspection, diagnosis, and strategy revision.
Z.ai
A GLM-5 technical report centered on agentic engineering, combining a more efficient long-context architecture with asynchronous reinforcement-learning infrastructure and agent RL for complex, long-horizon software tasks.
A working framework
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Read the analysis →RL Scaling studies what happens when reinforcement learning systems grow across compute, data, model capacity, rollout volume, task diversity, and feedback quality.
→02Reinforcement Learning with Verifiable Rewards uses outcomes that can be checked by programs, tests, simulators, or formal rules.
→03Group Relative Policy Optimization and related methods update a language-model policy using rewards from sampled response groups rather than a separately learned value critic.
→04Agentic RL applies reinforcement learning to systems that plan, use tools, interact with software, and act over long horizons.
→05An RL environment defines the tasks, observations, actions, transitions, and outcomes through which a policy gains experience.
→06Rewards translate outcomes and preferences into the signal used to improve a policy.
→07Post-training turns a pretrained model into a useful system through supervised fine-tuning, preference optimization, reinforcement learning, distillation, and targeted data generation.
→08Scaling laws describe empirical relationships between resources and performance.
→RL Scaling is an independent research publication. We follow reinforcement learning, post-training, and agents, with an emphasis on evidence, limitations, and the connections between ideas.
About RL Scaling →