Research index
Source-linked research notes with original summaries and a clear view of why each development matters for scaling intelligent systems.
Zhejiang University
KnowGPO groups search-agent steps by accumulated task-relevant knowledge and rewards newly acquired facts. Controlled Qwen3-8B comparisons show gains over GRPO on four evaluation splits, but the method requires reference answers, successful training trajectories, and an external LLM verifier. Several narrative claims in the first version disagree with its tables.
Tsinghua University
ActionOPD distills black-box agent teachers by rewarding semantic agreement between sampled student and teacher actions. It needs teacher generation rather than token probabilities or aligned tokenizers. Experiments report cross-benchmark transfer, alongside mixed success-versus-reliability results against token-level OPD.
Zhejiang University
A common evaluation compares ten agentic self-distillation methods with GRPO across Qwen3.5-2B, 4B, and 9B on four agent benchmarks. Across 132 training runs, gains depend strongly on the setting, final scores can conceal unstable learning, and additional training time often fails to improve performance.
Fin.ai
The paper probes whether reference-conditioned self-distillation actually locates and corrects reasoning errors. Generating a critique before scoring gives more error-specific feedback in the tested models, and small Knights & Knaves training experiments improve with a critique-conditioned term. Every critique has access to a reference solution.
Dalian Maritime University
A GSM8K study separates current-policy sample selection from progression between difficulty bands in GRPO. With Qwen2.5-1.5B-Instruct and LoRA, a staged curriculum reaching moderately harder problems improves test accuracy over persistent medium-difficulty selection, while a further difficulty increase reverses the gain across three seeds.
Tsinghua University
BranchOPD selects decision points using student uncertainty and teacher–student disagreement, samples student continuations from shared prefixes, and distills teacher feedback on the resulting branches. Protocol-boundary repair (PBR) separately addresses repetition and malformed response boundaries. Experiments cover mathematical reasoning and three agent benchmarks.
Alibaba Group
Zero-Sum Calibrated Policy Optimization (ZCPO) puts reference-policy drift into group-relative update coefficients instead of adding an independent KL loss. The paper analyzes seven potential KL–reward mismatches and reports improved math and long-context performance in its tested configurations.
Harbin Institute of Technology (Shenzhen)
GRPODropout selectively excludes high-probability positive-advantage rollouts from GRPO updates, then recenters retained advantages using old-policy probability weights. With the same generation budget, experiments on Qwen3 models report higher average benchmark accuracy and less entropy collapse than standard GRPO.
Xiaomi
Xiaomi’s MiMo-V2.6 report details large-scale mixed-task agentic RL: long-horizon batches, diverse environments and harnesses, and agentic grading beyond binary test rewards. It reports training-cost curves, router-stability experiments, infrastructure failures, and a subsequent multi-teacher distillation stage.
NVIDIA
NeMo-DCR transfers compressed weight changes between training and rollout clusters while reconstructing the same parameter and buffer bits as a dense refit. Cross-region benchmarks report large synchronization speedups, with separate small-scale training experiments testing recovery from interrupted updates.
Institute of Computing Technology, Chinese Academy of Sciences
VenusRL separates training, rollout generation, and tool environments, prioritizing trajectory groups that can unblock the next training batch. It combines priority-aware scheduling and KV-cache management with denser sandbox placement, reporting workload-dependent throughput gains on long-horizon software-agent training.
University of Illinois Urbana-Champaign
SURGE recombines updates from two checkpoints of the same RL run, retaining a leading spectral component from an accurate anchor and adding complementary donor updates. Across two 1.5B math histories and one 7B coding history, the constructed policies improve selected accuracy measures while generating fewer tokens than their anchors.
National University of Singapore
RISED uses shared behavioral rubrics to select rollout groups across agent environments and to condition an on-policy self-distillation teacher. It improves mean validation pass rate on two small-model backbones, while adding judge inference, teacher forward passes, and rollout oversampling.
University of Illinois Urbana-Champaign
DARS turns annotated task dependencies into signed step rewards: an unresolved mistake discounts dependent progress while independent work retains credit. Experiments cover household tasks, shopping, retrieval, and mathematics, with the clearest matched gain on a 1.5B ALFWorld agent.
Shanghai Jiao Tong University
MOPD-Router selects and weights teacher supervision at each token rather than assigning one domain teacher to a whole prompt. ExpertAlign uses each teacher’s shift from a shared pre-RL base to identify useful corrections, improving a nine-benchmark aggregate in four tested settings.
WeChat AI
WeEnv manages agentic RL environments across packaging, initialization, and resource provisioning. Composable layers and on-demand fetching reduce environment startup costs, while elastic CPU and memory quotas adapt to task demand.
Stanford University
A randomly initialized program generator learns through RL to produce synthetic byte sequences that train a separate learner. The resulting small models improve zero-shot prediction across natural-data modalities as self-play compute increases, without using natural data for gradient updates.
Shuai Dong
On-policy distillation can prepare a student for later RL even when its immediate accuracy gain is small. Controlled downstream recipes show stronger reported final performance from OPD starting policies, and a distillation objective ranking that reverses after RL.
Alibaba Group
CLOOPD reuses each teacher-scored batch for multiple student updates, separating the cost of acquiring supervision from the compute spent learning it. In one math-distillation setup, three actor passes nearly match a longer TOP-D run with fewer teacher tokens and allocated GPU-hours, while higher endpoint quality costs more compute.
Meta FAIR
WeightBridge maps trainer and rollout parameter layouts, then plans direct, balanced transfers of useful shards. On H100 deployments from 16B to 1T parameters, it sharply reduces GPU stalls during weight synchronization; the reported gains concern transfer overhead, not total RL training time.
Jie Zhao
Category-specific software-engineering experts alternate agentic RL with replay of their own successful trajectories, then transfer their gains into one student through label-routed on-policy distillation. The study tracks category regressions hidden by aggregate benchmark scores.
DeepSeek
DeepSeek Elastic Compute separates stateful agent execution from preemptible GPU training. Its sandbox platform combines on-demand image loading, composable filesystem layers, memory reclamation, and CPU scheduling to support large bursts of long-running agent workloads.
Yuxiao Yang
On-policy distillation can suppress a student’s stopping behavior when teacher and student favor different termination tokens. Grouping operationally equivalent EOS tokens at the probability level mitigates this mechanism, while a separate late-stage source of length inflation remains unresolved.
Together AI
Score centering removes an accumulating drift term caused by differences between rollout and training engines. Experiments test when this additive correction stabilizes quantized or stale-policy RL, and when it benefits from being combined with importance sampling.
Yan Yu
RetireOPD first trains a teacher with privileged task skills, then combines on-policy distillation with RL for a student that cannot access those skills. An adaptive rule retires the teacher once transfer stalls and the student reaches sufficient task competence.
AI Singapore
SEA-LION v4.8 adapts Nemotron models to Southeast Asian languages through continued pretraining, supervised fine-tuning, online on-policy distillation, and model merging. The report connects regional language performance to both data coverage and tokenizer constraints.
Hejia Geng
ScienceIDE turns scientific codebases and expert acceptance criteria into executable tasks, environments, and verifiers. Its experiments examine scientific software repair, supervised transfer, and RL with costly outcome feedback and long interaction histories.
Leon Bergen
Simple directions in model activations can flag reward-hacking behavior during agent evaluations. The study compares difference-of-means probes with LLM monitors, examines predictive signals before actions, and documents where detection still produces errors.
Dialpad Inc.
A controlled study compares SFT with LoRA, GRPO, and their sequential combination across six Qwen3 model sizes. SFT leads most in-distribution tool-calling settings, while RL’s cross-dataset advantage is smaller and dataset mixing remains consistently useful.
Michael Noukhovitch
Never Give Up reallocates rollout compute toward hard problems by quickly filtering easy groups and repeatedly sampling unsolved ones. The study links this adaptive allocation to better learning signals on difficult mathematical and coding tasks.
RadixArk
Miles v0.1 describes a full-stack post-training system spanning rollout generation, optimization, and weight synchronization. Its reference run demonstrates asynchronous terminal-agent RL with a 744B-parameter MoE model and makes the system’s supported configurations explicit.
Yang Li
RISE constructs a distillation teacher by extrapolating a model’s own RLVR training trajectory. Alternating reward-based updates with token-level distillation improves the tested reasoning and agent policies while reusing their sampled rollouts.
Shanghai Jiao Tong University
CUA-Universe turns desktop applications into shared-state GUI and CLI environments, then generates tasks and training trajectories. Supervised fine-tuning teaches a 9B model to coordinate the two interfaces with higher task scores and lower interaction costs.
Tsinghua University
An empirical study of how prompt difficulty and reasoning length affect on-policy distillation. Eight selected hard math problems approach the performance of a 17K-problem baseline, while repeated rollout generation and dense teacher supervision remain part of training.
Sogang University
A GRPO replay method that reuses past rollout groups according to their remaining learning value and compatibility with the current policy. Experiments examine when replay can replace costly fresh environment interactions in reasoning and agent training.
New York University
An empirical study of how to combine on-policy distillation with RLVR. Training with distillation first and RL second produces strong logic-reasoning gains, while the advantage over the strongest alternatives is less decisive on math.
RL Scaling
A practical framework for locating the binding constraint in an RL system across environments, rollout generation, reward quality, optimization, and evaluation.
Ziniu Li
An empirical decomposition of LLM reinforcement-learning batch scaling into sample-indexed learning behavior and systems throughput, with a decision rule for when a larger batch lowers wall-clock time-to-target.
Maciej Besta
A compute-centric systematization of RL-style reasoning-model post-training, covering PPO and GRPO as well as intra-model, inter-model, synchronous, and asynchronous parallelism.
Z.ai
An official GLM-5.3 release note describing a post-training-only update on the same base model as GLM-5.2, with expanded executable environments for complex coding, security, and long-horizon professional workflows.
Moonshot AI
A technical report on Kimi K3, a 2.8T-parameter sparse multimodal model with 104B active parameters and a one-million-token context window, post-trained with reinforcement learning across general, agentic, coding, and reasoning domains.
Yufei Li
A production-oriented technical report on a generative reasoning re-ranker trained with semantic-ID mid-training, teacher-trace distillation, on-policy distillation, and reinforcement learning from verifiable ranking rewards.
Sakana AI
A technical report on Fugu and Fugu-Ultra, language-model orchestrators trained to construct query-adaptive scaffolds for teams of heterogeneous LLM agents.
Z.ai
An official release of the 750B-A40B GLM-5.2 model, combining a one-million-token context, IndexShare sparse-attention reuse, and larger-scale agentic reinforcement learning for long-horizon tasks.
Ang Li
A model-family report on Ling-2.6 and Ring-2.6, combining architectural migration, long-context efficiency work, token-efficient reasoning objectives, and asynchronous agent reinforcement learning at trillion-parameter scale.
MiniMax
An official release of MiniMax-M3, a natively multimodal sparse model with about 428B total and 23B active parameters, a one-million-token context window, and agent-oriented coding and cowork capabilities.
MiniMax
A technical report on the MiniMax-M2 family, pairing a 229.9B-parameter sparse MoE with agent-generated, verifiable trajectories and Forge, a scalable reinforcement-learning system for long-horizon agents.
Cursor
A technical report on Composer 2, a specialized coding model trained through continued pretraining followed by large-scale reinforcement learning on long-horizon software-engineering tasks.
Z.ai
A GLM-5 technical report centered on agentic engineering, combining a more efficient long-context architecture with asynchronous reinforcement-learning infrastructure and agent RL for complex, long-horizon software tasks.
DAPO Team
DAPO presents an open reinforcement-learning system and a collection of training interventions aimed at improving large-scale reasoning optimization.
Kimi Team
A technical report on scaling multimodal language-model reinforcement learning with long-context training, policy optimization, and data-selection strategies.
OpenAI
An empirical study of how optimizing against a learned reward model can eventually reduce performance under a higher-quality reference signal.
OpenAI
The OpenAI Five report documents a distributed, continually trained self-play system for a long-horizon, imperfect-information team game.
OpenAI
PPO introduces clipped and adaptive objectives intended to make policy-gradient optimization simpler to implement while limiting excessively large policy updates.