TL;DR
Checkpoint handoff separates two sources of agentic RL gains: reaching useful states and completing tasks from those states. Replaying histories across SFT and RL checkpoints reveals an interaction that endpoint success alone cannot attribute.
Why it matters
An agent changes its own future inputs through earlier actions. Comparing only final success mixes better state preparation with better decisions from a given state, making it difficult to identify what additional RL training actually improves.
Key findings
- 01
The protocol crosses two roles: a reacher produces a history or state, and a solver continues it. Evaluating all four SFT/RL combinations distinguishes the effect of the continuation policy from the states selected by the earlier policy.
- 02
TravelPlanner experiments use released Agent-STAR checkpoints at 1.5B, 3B, and 7B on 180 tasks per scale. ALFWorld uses SkillRL 7B checkpoints with 30 prespecified unseen tasks and four paired seeds, plus a separate seen-split sample.
- 03
In ALFWorld, the environment certifies that handoff states are exactly two actions from success. On the unseen sample, the SFT reacher reaches this frontier in 13.3% of trajectories versus 82.5% for the RL reacher. From the same 115 cloned frontier states, solver completion is 56.5% for SFT and 95.7% for RL.
- 04
The reacher–solver interaction is positive in all five reported conditions, with task-level paired-bootstrap intervals above zero. This means the RL solver’s contribution to endpoint success depends on which checkpoint prepared its inputs; it is not a fixed, context-independent increment.
Evidence & limits
The experiments cover two Qwen-family training pipelines and two benchmarks. The ALFWorld component analysis uses a small prespecified task sample and a two-action continuation horizon, not arbitrary long-horizon finishing behavior.
Exact frontier verification relies on enumerable admissible actions and reproducible state replay. TravelPlanner instead uses a recorded-history cut before the final model call, so it does not provide the same environment-certified distance decomposition.
Frontier arrival and completion from selected cloned states are different metrics from ordinary end-to-end task success. The 95.7% solver figure should not be presented as a general ALFWorld success rate.
This is an evaluation protocol applied to released checkpoints, not a new training algorithm or a demonstrated compute-saving intervention. The apparent decrease in interaction with model size has overlapping intervals and is not established as a scaling trend.
Scaling dimensions
Models, methods & benchmarks
- Models
- Agent-STAR-1.5B, Agent-STAR-3B, Agent-STAR-7B, SkillRL-7B
- Algorithms
- Checkpoint handoff, State replay, Paired evaluation
- Benchmarks
- TravelPlanner, ALFWorld
Topics