Research index
Source-linked research notes with original summaries and a clear view of why each development matters for scaling intelligent systems.
Zhejiang University
KnowGPO groups search-agent steps by accumulated task-relevant knowledge and rewards newly acquired facts. Controlled Qwen3-8B comparisons show gains over GRPO on four evaluation splits, but the method requires reference answers, successful training trajectories, and an external LLM verifier. Several narrative claims in the first version disagree with its tables.
Tsinghua University
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
Zhejiang University
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
Fin.ai
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
Dalian Maritime University
A GSM8K study separates current-policy sample selection from progression between difficulty bands in GRPO. With Qwen2.5-1.5B-Instruct and LoRA, a staged curriculum reaching moderately harder problems improves test accuracy over persistent medium-difficulty selection, while a further difficulty increase reverses the gain across three seeds.
Tsinghua University
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
Alibaba Group
Zero-Sum Calibrated Policy Optimization (ZCPO) puts reference-policy drift into group-relative update coefficients instead of adding an independent KL loss. The paper analyzes seven potential KL–reward mismatches and reports improved math and long-context performance in its tested configurations.
Harbin Institute of Technology (Shenzhen)
GRPODropout selectively excludes high-probability positive-advantage rollouts from GRPO updates, then recenters retained advantages using old-policy probability weights. With the same generation budget, experiments on Qwen3 models report higher average benchmark accuracy and less entropy collapse than standard GRPO.
Xiaomi
Xiaomi’s MiMo-V2.6 report details large-scale mixed-task agentic RL: long-horizon batches, diverse environments and harnesses, and agentic grading beyond binary test rewards. It reports training-cost curves, router-stability experiments, infrastructure failures, and a subsequent multi-teacher distillation stage.
National University of Singapore
RISED uses shared behavioral rubrics to select rollout groups across agent environments and to condition an on-policy self-distillation teacher. It improves mean validation pass rate on two small-model backbones, while adding judge inference, teacher forward passes, and rollout oversampling.
University of Illinois Urbana-Champaign
DARS turns annotated task dependencies into signed step rewards: an unresolved mistake discounts dependent progress while independent work retains credit. Experiments cover household tasks, shopping, retrieval, and mathematics, with the clearest matched gain on a 1.5B ALFWorld agent.
Li, Shuyue Stella
Verifiable Visual Rewards derives image prompts and deterministic constraint checks from the same geometric scene specification. Flow-GRPO training on simple synthetic scenes improves instruction following in Stable Diffusion 3.5 Medium and transfers to natural-image prompts, with measurable tradeoffs.
Shanghai Jiao Tong University
MOPD-Router selects and weights teacher supervision at each token rather than assigning one domain teacher to a whole prompt. ExpertAlign uses each teacher’s shift from a shared pre-RL base to identify useful corrections, improving a nine-benchmark aggregate in four tested settings.
Stanford University
A randomly initialized program generator learns through RL to produce synthetic byte sequences that train a separate learner. The resulting small models improve zero-shot prediction across natural-data modalities as self-play compute increases, without using natural data for gradient updates.
Shuai Dong
On-policy distillation can prepare a student for later RL even when its immediate accuracy gain is small. Controlled downstream recipes show stronger reported final performance from OPD starting policies, and a distillation objective ranking that reverses after RL.
Kuaishou
KwaiMind combines diffusion preference optimization, task-specific online RL, and on-policy distillation for commercial image editing. Dedicated rewards cover predicted click-through rate, text rendering, and product consistency; the report separates offline reward-based gains from an online image-selection experiment.
Alibaba Group
CLOOPD reuses each teacher-scored batch for multiple student updates, separating the cost of acquiring supervision from the compute spent learning it. In one math-distillation setup, three actor passes nearly match a longer TOP-D run with fewer teacher tokens and allocated GPU-hours, while higher endpoint quality costs more compute.
Jie Zhao
Category-specific software-engineering experts alternate agentic RL with replay of their own successful trajectories, then transfer their gains into one student through label-routed on-policy distillation. The study tracks category regressions hidden by aggregate benchmark scores.
Xiaomi
Xiaomi’s CodeMidas turns implemented functionality in existing codebases into executable coding RL tasks. Training MiMo-V2.5 on 5,545 filtered environments improves five external benchmarks, while a 3,000-task filtered subset beats an 8,000-task unfiltered baseline on three evaluations.
Juzheng Zhang
ActObs includes environment observations in the supervised loss on existing agent trajectories. The resulting initialization supports different exploration behavior during subsequent RL, with gains that depend on model size and evaluation sampling budget.
Yan Yu
RetireOPD first trains a teacher with privileged task skills, then combines on-policy distillation with RL for a student that cannot access those skills. An adaptive rule retires the teacher once transfer stalls and the student reaches sufficient task competence.
AI Singapore
SEA-LION v4.8 adapts Nemotron models to Southeast Asian languages through continued pretraining, supervised fine-tuning, online on-policy distillation, and model merging. The report connects regional language performance to both data coverage and tokenizer constraints.
Hejia Geng
ScienceIDE turns scientific codebases and expert acceptance criteria into executable tasks, environments, and verifiers. Its experiments examine scientific software repair, supervised transfer, and RL with costly outcome feedback and long interaction histories.
Dialpad Inc.
A controlled study compares SFT with LoRA, GRPO, and their sequential combination across six Qwen3 model sizes. SFT leads most in-distribution tool-calling settings, while RL’s cross-dataset advantage is smaller and dataset mixing remains consistently useful.
Michael Noukhovitch
Never Give Up reallocates rollout compute toward hard problems by quickly filtering easy groups and repeatedly sampling unsolved ones. The study links this adaptive allocation to better learning signals on difficult mathematical and coding tasks.
kaivu
A research-methods proposal for studying when models misbehave in realistic use. It starts from recorded failures, varies environment conditions or the policy, and asks how those observations can support estimates of behavior in a target task distribution.
Yang Li
RISE constructs a distillation teacher by extrapolating a model’s own RLVR training trajectory. Alternating reward-based updates with token-level distillation improves the tested reasoning and agent policies while reusing their sampled rollouts.
Shanghai Jiao Tong University
CUA-Universe turns desktop applications into shared-state GUI and CLI environments, then generates tasks and training trajectories. Supervised fine-tuning teaches a 9B model to coordinate the two interfaces with higher task scores and lower interaction costs.
Tsinghua University
An empirical study of how prompt difficulty and reasoning length affect on-policy distillation. Eight selected hard math problems approach the performance of a 17K-problem baseline, while repeated rollout generation and dense teacher supervision remain part of training.
Qwen Team, Alibaba Group
Terminal-Universe reconstructs reusable terminal environments from recorded agent trajectories, then generates new tasks within projects, across related codebases, and over multiple user rounds. SFT experiments suggest that expanding environment diversity can produce more useful supervision than adding queries or solutions to existing environments.
Sogang University
A GRPO replay method that reuses past rollout groups according to their remaining learning value and compatibility with the current policy. Experiments examine when replay can replace costly fresh environment interactions in reasoning and agent training.
New York University
An empirical study of how to combine on-policy distillation with RLVR. Training with distillation first and RL second produces strong logic-reasoning gains, while the advantage over the strongest alternatives is less decisive on math.
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Xingyu Shen
A method for preserving reasoning diversity during RLVR by conditioning the target model on partial trajectories produced by a smaller, weaker model.
Steven Byrnes
A community argument that RLVR is operationally important but that most language-model knowledge and available reasoning strategies still originate in pretraining and supervised imitation.
Yufei Li
A production-oriented technical report on a generative reasoning re-ranker trained with semantic-ID mid-training, teacher-trace distillation, on-policy distillation, and reinforcement learning from verifiable ranking rewards.
CarolusRenniusVitellius
A concise reading map of weight-level and behavioral evidence on whether RLVR mainly reweights existing capabilities or creates new reasoning mechanisms.
DeepSeek-AI
DeepSeek-R1 documents a reasoning-model training pipeline centered on reinforcement learning, including a zero-style RL stage and a staged route that combines supervised data with RL.
Kimi Team
A technical report on scaling multimodal language-model reinforcement learning with long-context training, policy optimization, and data-selection strategies.
DeepSeek-AI
DeepSeekMath combines a math-focused data pipeline with supervised fine-tuning and reinforcement learning, and introduces Group Relative Policy Optimization.
Meta FAIR
This work studies language models that generate candidate responses and also provide the preference signal used to improve subsequent iterations.
Google DeepMind
ReST alternates offline sample generation with policy improvement, allowing generated data to be reused rather than coupling every update to fresh online rollouts.
OpenAI
A comparison of outcome supervision and process supervision for mathematical reasoning, centered on whether feedback should evaluate only the final answer or intermediate steps as well.
Stanford University
DPO reformulates preference optimization as a direct classification-style objective, avoiding an explicit reward-model training stage and online policy optimization.
Guanzhi Wang
Voyager combines an automatic curriculum, an executable skill library, and iterative prompting to build an open-ended Minecraft agent without model parameter updates.
Google DeepMind
DreamerV3 presents a world-model-based reinforcement-learning algorithm designed to work across varied domains with a shared configuration.
Anthropic
Constitutional AI uses written principles and model-generated critiques and preferences to reduce direct dependence on human harmlessness labels during alignment training.
NVIDIA
MineDojo combines a Minecraft simulation suite, a broad task set, and internet-scale multimodal knowledge for research on generalist embodied agents.
OpenAI
The InstructGPT work combines demonstrations, preference comparisons, reward modeling, and PPO to align language-model behavior with user instructions.
Michael Dennis
This work frames task-distribution design as an optimization problem and introduces PAIRED, where an environment generator uses regret to create solvable but challenging curricula.
OpenAI
A large-scale study of training summarization models from human comparisons using reward modeling and reinforcement learning.
OpenAI
A foundational study of learning a reward model from pairwise human preferences and using that learned reward to train an RL agent.