Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

TL;DR

ActObs includes environment observations in the supervised loss on existing agent trajectories. The resulting initialization supports different exploration behavior during subsequent RL, with gains that depend on model size and evaluation sampling budget.

Why it matters

The supervision mask used before RL can shape which behaviors the policy later explores. Learning from the consequences of actions offers another way to use existing trajectories without collecting more demonstrations.

Key findings

1. 01

   The primary comparison uses the same 50,000 terminal trajectories, totaling 0.71B tokens, with Qwen3-4B and Qwen3-8B. ActionSFT predicts only actions; ActObs also predicts observations. Both then receive 135 GRPO steps on 2,392 separate containerized tasks.

2. 02

   On Terminal-Bench 2.0, ActObs followed by GRPO reaches 7.2% pass@1 at 4B versus 5.6% from action-only SFT followed by GRPO. At 8B, pass@16 rises from 23.6% to 27.0%, while pass@1 falls from 12.3% to 11.0%.

3. 03

   On 225 unseen aider-polyglot code-editing tasks, 4B pass@1 rises from 9.7% to 13.9%, a 4.2-percentage-point improvement. This evaluates transfer beyond the terminal tasks used for training.

4. 04

   Training analyses associate joint supervision with better retained observation prediction, higher entropy during RL, and less policy movement from the SFT checkpoint. The standard ActObs-to-GRPO recipe adds observation supervision during SFT, not an observation-prediction loss during RL.

Evidence & limits

• —

  The 8B result trades single-attempt reliability for higher repeated-sampling success. It is not a uniform improvement at every inference budget.

• —

  Reported bootstrap errors resample attempts while holding the task set fixed. They capture finite-attempt evaluation variation, not variation across independent training seeds or new task populations.

• —

  Experiments cover two Qwen3 scales and terminal/code-editing benchmarks. Reusing the same input tokens and forward passes does not establish a universal end-to-end training speedup or gains in other agent domains.

Scaling dimensions

Models, methods & benchmarks

Models

Qwen3-4B, Qwen3-8B

Algorithms

ActObs, Supervised Fine-Tuning, GRPO

Benchmarks

Terminal-Bench 2.0, aider-polyglot

Topics

Source materials

• Full paper and experimental details ↗