TL;DR
ScienceIDE turns scientific codebases and expert acceptance criteria into executable tasks, environments, and verifiers. Its experiments examine scientific software repair, supervised transfer, and RL with costly outcome feedback and long interaction histories.
Why it matters
Environment supply is a distinct constraint on scaling agent training. Scientific repositories offer grounded tasks, but their usefulness depends on expert verification, execution cost, and an objective that does not punish difficult attempts merely for reaching a budget limit.
Key findings
- 01
The reported inventory contains 64 environments from 27 scientific codebases, 2,812 manufactured tasks, and 1,076 executable scientific checks. Repair and implementation account for almost all tasks; the hard evaluation subset contains 85 tasks from 18 environments.
- 02
Expert-defined contracts specify scientific observables and acceptance tolerances. Task factories reuse these decisions, while candidate tasks still require executable validation against scientific reference outputs.
- 03
RL experiments train Qwen3.5-4B on LAPS and MITgcm-biogeo. Budget-truncated trajectories remain in the group reward baseline but are masked out of the policy loss, addressing a failure mode in which penalizing unfinished attempts encourages shorter, less useful behavior.
- 04
After 30 training steps, mean held-out shaped verifier reward rises from 0.357 to 0.857 on 14 LAPS tasks and from 0.286 to 0.571 on 21 MITgcm-biogeo tasks. Both evaluations use matched harnesses and defect-location hints within the training environments.
Evidence & limits
The RL results are shaped rewards on small, hinted, within-environment held-out sets, not strict success rates or demonstrated transfer to unseen scientific codebases. Between-seed variance is not estimated.
The evidence concerns reference-verifiable repair and implementation, mainly in computational physics and geoscience. Open-ended scientific discovery and the proposed 1,000-environment release remain outside the demonstrated scope.
Task factories amortize expert work but do not eliminate it. Generated variants may share code paths and assumptions; task count alone does not measure independent scientific coverage.
Reference checks and calibrated tolerances cannot certify behavior outside their scope. Independent external audits and adversarial reward-hacking evaluations remain outstanding, and the authors describe residual contamination and selection limitations.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3.5-4B, PhAI-IDE-72B, PhAI-IDE-9B, PhAI-IDE-4B
- Algorithms
- Supervised Fine-Tuning, Group-relative policy optimization, Truncation masking
- Benchmarks
- ScienceIDE-Hard, LAPS, MITgcm-biogeo
Topics