Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

TL;DR

A work-in-progress study of long-horizon agent reinforcement learning that uses privileged process supervision to redistribute verified trajectory-level credit across executable actions.

Why it matters

Outcome rewards tell an agent whether a trajectory succeeded but not which decisions deserved credit. The proposed TASPO method is an attempt to make that signal more local without letting an auxiliary supervisor override the verified outcome.

Key findings

1. 01

   TASPO keeps the verified outcome responsible for the update direction and average scale while privileged information only changes how credit is distributed across actions.

2. 02

   The authors report a 10.6% improvement over GRPO across three agentic benchmarks and better generalization to unseen tasks.

3. 03

   The paper is explicitly labeled work in progress, so the result should be treated as early evidence pending replication.

Scaling dimensions

Models, methods & benchmarks

Algorithms

TASPO, GRPO, On-policy self-distillation

Topics