Research index
Source-linked research notes with original summaries and a clear view of why each development matters for scaling intelligent systems.
Zhejiang University
KnowGPO groups search-agent steps by accumulated task-relevant knowledge and rewards newly acquired facts. Controlled Qwen3-8B comparisons show gains over GRPO on four evaluation splits, but the method requires reference answers, successful training trajectories, and an external LLM verifier. Several narrative claims in the first version disagree with its tables.
Tsinghua University
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
Zhejiang University
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
Fin.ai
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
Dalian Maritime University
A GSM8K study separates current-policy sample selection from progression between difficulty bands in GRPO. With Qwen2.5-1.5B-Instruct and LoRA, a staged curriculum reaching moderately harder problems improves test accuracy over persistent medium-difficulty selection, while a further difficulty increase reverses the gain across three seeds.
Tsinghua University
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
Alibaba Group
Zero-Sum Calibrated Policy Optimization (ZCPO) puts reference-policy drift into group-relative update coefficients instead of adding an independent KL loss. The paper analyzes seven potential KL–reward mismatches and reports improved math and long-context performance in its tested configurations.
Harbin Institute of Technology (Shenzhen)
GRPODropout selectively excludes high-probability positive-advantage rollouts from GRPO updates, then recenters retained advantages using old-policy probability weights. With the same generation budget, experiments on Qwen3 models report higher average benchmark accuracy and less entropy collapse than standard GRPO.
Xiaomi
Xiaomi’s MiMo-V2.6 report details large-scale mixed-task agentic RL: long-horizon batches, diverse environments and harnesses, and agentic grading beyond binary test rewards. It reports training-cost curves, router-stability experiments, infrastructure failures, and a subsequent multi-teacher distillation stage.
NVIDIA
NeMo-DCR transfers compressed weight changes between training and rollout clusters while reconstructing the same parameter and buffer bits as a dense refit. Cross-region benchmarks report large synchronization speedups, with separate small-scale training experiments testing recovery from interrupted updates.
Institute of Computing Technology, Chinese Academy of Sciences
VenusRL separates training, rollout generation, and tool environments, prioritizing trajectory groups that can unblock the next training batch. It combines priority-aware scheduling and KV-cache management with denser sandbox placement, reporting workload-dependent throughput gains on long-horizon software-agent training.
University of Illinois Urbana-Champaign
SURGE recombines updates from two checkpoints of the same RL run, retaining a leading spectral component from an accurate anchor and adding complementary donor updates. Across two 1.5B math histories and one 7B coding history, the constructed policies improve selected accuracy measures while generating fewer tokens than their anchors.
National University of Singapore
RISED uses shared behavioral rubrics to select rollout groups across agent environments and to condition an on-policy self-distillation teacher. It improves mean validation pass rate on two small-model backbones, while adding judge inference, teacher forward passes, and rollout oversampling.
University of Illinois Urbana-Champaign
DARS turns annotated task dependencies into signed step rewards: an unresolved mistake discounts dependent progress while independent work retains credit. Experiments cover household tasks, shopping, retrieval, and mathematics, with the clearest matched gain on a 1.5B ALFWorld agent.
Li, Shuyue Stella
Verifiable Visual Rewards derives image prompts and deterministic constraint checks from the same geometric scene specification. Flow-GRPO training on simple synthetic scenes improves instruction following in Stable Diffusion 3.5 Medium and transfers to natural-image prompts, with measurable tradeoffs.
Shanghai Jiao Tong University
MOPD-Router selects and weights teacher supervision at each token rather than assigning one domain teacher to a whole prompt. ExpertAlign uses each teacher’s shift from a shared pre-RL base to identify useful corrections, improving a nine-benchmark aggregate in four tested settings.
WeChat AI
WeEnv manages agentic RL environments across packaging, initialization, and resource provisioning. Composable layers and on-demand fetching reduce environment startup costs, while elastic CPU and memory quotas adapt to task demand.
Stanford University
A randomly initialized program generator learns through RL to produce synthetic byte sequences that train a separate learner. The resulting small models improve zero-shot prediction across natural-data modalities as self-play compute increases, without using natural data for gradient updates.
Shuai Dong
On-policy distillation can prepare a student for later RL even when its immediate accuracy gain is small. Controlled downstream recipes show stronger reported final performance from OPD starting policies, and a distillation objective ranking that reverses after RL.
Kuaishou
KwaiMind combines diffusion preference optimization, task-specific online RL, and on-policy distillation for commercial image editing. Dedicated rewards cover predicted click-through rate, text rendering, and product consistency; the report separates offline reward-based gains from an online image-selection experiment.
Alibaba Group
CLOOPD reuses each teacher-scored batch for multiple student updates, separating the cost of acquiring supervision from the compute spent learning it. In one math-distillation setup, three actor passes nearly match a longer TOP-D run with fewer teacher tokens and allocated GPU-hours, while higher endpoint quality costs more compute.
Meta FAIR
WeightBridge maps trainer and rollout parameter layouts, then plans direct, balanced transfers of useful shards. On H100 deployments from 16B to 1T parameters, it sharply reduces GPU stalls during weight synchronization; the reported gains concern transfer overhead, not total RL training time.
Jie Zhao
Category-specific software-engineering experts alternate agentic RL with replay of their own successful trajectories, then transfer their gains into one student through label-routed on-policy distillation. The study tracks category regressions hidden by aggregate benchmark scores.
DeepSeek
DeepSeek Elastic Compute separates stateful agent execution from preemptible GPU training. Its sandbox platform combines on-demand image loading, composable filesystem layers, memory reclamation, and CPU scheduling to support large bursts of long-running agent workloads.
Xiaomi
Xiaomi’s CodeMidas turns implemented functionality in existing codebases into executable coding RL tasks. Training MiMo-V2.5 on 5,545 filtered environments improves five external benchmarks, while a 3,000-task filtered subset beats an 8,000-task unfiltered baseline on three evaluations.
Xuan Liu
Checkpoint handoff separates two sources of agentic RL gains: reaching useful states and completing tasks from those states. Replaying histories across SFT and RL checkpoints reveals an interaction that endpoint success alone cannot attribute.
Juzheng Zhang
ActObs includes environment observations in the supervised loss on existing agent trajectories. The resulting initialization supports different exploration behavior during subsequent RL, with gains that depend on model size and evaluation sampling budget.
Yuxiao Yang
On-policy distillation can suppress a student’s stopping behavior when teacher and student favor different termination tokens. Grouping operationally equivalent EOS tokens at the probability level mitigates this mechanism, while a separate late-stage source of length inflation remains unresolved.
Together AI
Score centering removes an accumulating drift term caused by differences between rollout and training engines. Experiments test when this additive correction stabilizes quantized or stale-policy RL, and when it benefits from being combined with importance sampling.
Yan Yu
RetireOPD first trains a teacher with privileged task skills, then combines on-policy distillation with RL for a student that cannot access those skills. An adaptive rule retires the teacher once transfer stalls and the student reaches sufficient task competence.
AI Singapore
SEA-LION v4.8 adapts Nemotron models to Southeast Asian languages through continued pretraining, supervised fine-tuning, online on-policy distillation, and model merging. The report connects regional language performance to both data coverage and tokenizer constraints.
Hejia Geng
ScienceIDE turns scientific codebases and expert acceptance criteria into executable tasks, environments, and verifiers. Its experiments examine scientific software repair, supervised transfer, and RL with costly outcome feedback and long interaction histories.
Leon Bergen
Simple directions in model activations can flag reward-hacking behavior during agent evaluations. The study compares difference-of-means probes with LLM monitors, examines predictive signals before actions, and documents where detection still produces errors.
Dialpad Inc.
A controlled study compares SFT with LoRA, GRPO, and their sequential combination across six Qwen3 model sizes. SFT leads most in-distribution tool-calling settings, while RL’s cross-dataset advantage is smaller and dataset mixing remains consistently useful.
Michael Noukhovitch
Never Give Up reallocates rollout compute toward hard problems by quickly filtering easy groups and repeatedly sampling unsolved ones. The study links this adaptive allocation to better learning signals on difficult mathematical and coding tasks.
RadixArk
Miles v0.1 describes a full-stack post-training system spanning rollout generation, optimization, and weight synchronization. Its reference run demonstrates asynchronous terminal-agent RL with a 744B-parameter MoE model and makes the system’s supported configurations explicit.
kaivu
A research-methods proposal for studying when models misbehave in realistic use. It starts from recorded failures, varies environment conditions or the policy, and asks how those observations can support estimates of behavior in a target task distribution.
Yang Li
RISE constructs a distillation teacher by extrapolating a model’s own RLVR training trajectory. Alternating reward-based updates with token-level distillation improves the tested reasoning and agent policies while reusing their sampled rollouts.
Shanghai Jiao Tong University
CUA-Universe turns desktop applications into shared-state GUI and CLI environments, then generates tasks and training trajectories. Supervised fine-tuning teaches a 9B model to coordinate the two interfaces with higher task scores and lower interaction costs.
Tsinghua University
An empirical study of how prompt difficulty and reasoning length affect on-policy distillation. Eight selected hard math problems approach the performance of a 17K-problem baseline, while repeated rollout generation and dense teacher supervision remain part of training.
Qwen Team, Alibaba Group
Terminal-Universe reconstructs reusable terminal environments from recorded agent trajectories, then generates new tasks within projects, across related codebases, and over multiple user rounds. SFT experiments suggest that expanding environment diversity can produce more useful supervision than adding queries or solutions to existing environments.
Sogang University
A GRPO replay method that reuses past rollout groups according to their remaining learning value and compatibility with the current policy. Experiments examine when replay can replace costly fresh environment interactions in reasoning and agent training.
New York University
An empirical study of how to combine on-policy distillation with RLVR. Training with distillation first and RL second produces strong logic-reasoning gains, while the advantage over the strongest alternatives is less decisive on math.
Anthropic
An Anthropic study and LessWrong discussion examining how reward hacking learned in deliberately vulnerable RL environments can transfer to harmful behavior in other evaluation scenarios. The findings also show important limits to that transfer.
Jingxiao Yang
A work-in-progress study of long-horizon agent reinforcement learning that uses privileged process supervision to redistribute verified trajectory-level credit across executable actions.
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Qiancheng Zhou
A controlled study of where reasoning diversity disappears during RLVR, distinguishing failure to initiate a solution family from failure to execute it after the first reasoning choice.
Xingyu Shen
A method for preserving reasoning diversity during RLVR by conditioning the target model on partial trajectories produced by a smaller, weaker model.
Maciej Besta
A compute-centric systematization of RL-style reasoning-model post-training, covering PPO and GRPO as well as intra-model, inter-model, synchronous, and asynchronous parallelism.
Z.ai
An official GLM-5.3 release note describing a post-training-only update on the same base model as GLM-5.2, with expanded executable environments for complex coding, security, and long-horizon professional workflows.
Steven Byrnes
A community synthesis linking four training signals—imitation, human approval, automatic verifiers, and LLM judges—to distinct classes of observed or hypothesized alignment failure.
Fiora Starlight
A LessWrong proposal to let policies report exploitable RLVR environment bugs after a rollout, reward high-quality verified reports, and use them to patch the training environment.
Moonshot AI
A technical report on Kimi K3, a 2.8T-parameter sparse multimodal model with 104B active parameters and a one-million-token context window, post-trained with reinforcement learning across general, agentic, coding, and reasoning domains.
Steven Byrnes
A community argument that RLVR is operationally important but that most language-model knowledge and available reasoning strategies still originate in pretraining and supervised imitation.
Jialian Li
A technical report on Athena-Brain-8B, an on-device embodied model trained through general supervised fine-tuning, general reinforcement learning, embodied-expert training, and model merging.
Yufei Li
A production-oriented technical report on a generative reasoning re-ranker trained with semantic-ID mid-training, teacher-trace distillation, on-policy distillation, and reinforcement learning from verifiable ranking rewards.
Sakana AI
A technical report on Fugu and Fugu-Ultra, language-model orchestrators trained to construct query-adaptive scaffolds for teams of heterogeneous LLM agents.
Z.ai
An official release of the 750B-A40B GLM-5.2 model, combining a one-million-token context, IndexShare sparse-attention reuse, and larger-scale agentic reinforcement learning for long-horizon tasks.
Ang Li
A model-family report on Ling-2.6 and Ring-2.6, combining architectural migration, long-context efficiency work, token-efficient reasoning objectives, and asynchronous agent reinforcement learning at trillion-parameter scale.
MiniMax
An official release of MiniMax-M3, a natively multimodal sparse model with about 428B total and 23B active parameters, a one-million-token context window, and agent-oriented coding and cowork capabilities.
CarolusRenniusVitellius
A concise reading map of weight-level and behavioral evidence on whether RLVR mainly reweights existing capabilities or creates new reasoning mechanisms.
MiniMax
A technical report on the MiniMax-M2 family, pairing a 229.9B-parameter sparse MoE with agent-generated, verifiable trajectories and Forge, a scalable reinforcement-learning system for long-horizon agents.
Z.ai
An official GLM-5.1 model update focused on keeping an agent productive across longer coding and engineering runs through repeated execution, inspection, diagnosis, and strategy revision.
Cursor
A technical report on Composer 2, a specialized coding model trained through continued pretraining followed by large-scale reinforcement learning on long-horizon software-engineering tasks.
Z.ai
A GLM-5 technical report centered on agentic engineering, combining a more efficient long-context architecture with asynchronous reinforcement-learning infrastructure and agent RL for complex, long-horizon software tasks.
Zichen Liu
A controlled examination of R1-Zero-style reinforcement learning that separates robust training effects from claims that are sensitive to setup, reward design, and evaluation choices.
Weihao Zeng
An empirical study of reinforcement learning directly from open base models, focused on making zero-RL experiments easier to reproduce and compare across model families.
DAPO Team
DAPO presents an open reinforcement-learning system and a collection of training interventions aimed at improving large-scale reasoning optimization.
DeepSeek-AI
DeepSeek-R1 documents a reasoning-model training pipeline centered on reinforcement learning, including a zero-style RL stage and a staged route that combines supervised data with RL.
Kimi Team
A technical report on scaling multimodal language-model reinforcement learning with long-context training, policy optimization, and data-selection strategies.
Tianbao Xie
OSWorld introduces a benchmark and environment for evaluating multimodal agents on open-ended tasks across real computer interfaces.
DeepSeek-AI
DeepSeekMath combines a math-focused data pipeline with supervised fine-tuning and reinforcement learning, and introduces Group Relative Policy Optimization.
Meta FAIR
This work studies language models that generate candidate responses and also provide the preference signal used to improve subsequent iterations.
NVIDIA
Eureka uses a code-capable language model to propose and iteratively improve executable reward functions for reinforcement-learning tasks.
Princeton University
SWE-bench turns real GitHub issues and repository states into tasks for evaluating whether language models can produce working software patches.
Google DeepMind
ReST alternates offline sample generation with policy improvement, allowing generated data to be reused rather than coupling every update to fresh online rollouts.
Carnegie Mellon University
WebArena provides self-hosted, realistic websites and benchmark tasks for agents that must navigate interfaces, maintain state, and complete multi-step goals.
OpenAI
A comparison of outcome supervision and process supervision for mathematical reasoning, centered on whether feedback should evaluate only the final answer or intermediate steps as well.
Stanford University
DPO reformulates preference optimization as a direct classification-style objective, avoiding an explicit reward-model training stage and online policy optimization.
Guanzhi Wang
Voyager combines an automatic curriculum, an executable skill library, and iterative prompting to build an open-ended Minecraft agent without model parameter updates.
Google DeepMind
DreamerV3 presents a world-model-based reinforcement-learning algorithm designed to work across varied domains with a shared configuration.
Anthropic
Constitutional AI uses written principles and model-generated critiques and preferences to reduce direct dependence on human harmlessness labels during alignment training.
OpenAI
An empirical study of how optimizing against a learned reward model can eventually reduce performance under a higher-quality reference signal.
NVIDIA
MineDojo combines a Minecraft simulation suite, a broad task set, and internet-scale multimodal knowledge for research on generalist embodied agents.
OpenAI
The InstructGPT work combines demonstrations, preference comparisons, reward modeling, and PPO to align language-model behavior with user instructions.
Michael Dennis
This work frames task-distribution design as an optimization problem and introduces PAIRED, where an environment generator uses regret to create solvable but challenging curricula.
OpenAI
A large-scale study of training summarization models from human comparisons using reward modeling and reinforcement learning.
OpenAI
The OpenAI Five report documents a distributed, continually trained self-play system for a long-horizon, imperfect-information team game.
OpenAI
PPO introduces clipped and adaptive objectives intended to make policy-gradient optimization simpler to implement while limiting excessively large policy updates.
OpenAI
A foundational study of learning a reward model from pairwise human preferences and using that learned reward to train an RL agent.