TL;DR
Terminal-Universe reconstructs reusable terminal environments from recorded agent trajectories, then generates new tasks within projects, across related codebases, and over multiple user rounds. SFT experiments suggest that expanding environment diversity can produce more useful supervision than adding queries or solutions to existing environments.
Why it matters
Existing agent traces can supply the next generation of executable training environments. The paper makes environment diversity a concrete budget choice, with implications for the task distributions and verifiers that agent training depends on.
Key findings
- 01
Deterministic replay recovers file contents from before the source agent changed them and excludes files it created to solve the task. A completion agent restores missing context without implementing the requested change. A model judge retains 37,273 environments with sufficient context for their recovered tasks.
- 02
The main terminal SFT corpus contains 31,977 demonstrations and about 1.42 billion tokens, generated with Qwen3.7-Max. After two SFT epochs, Qwen3.5-27B improves from 46.2% to 58.1% on Terminal-Bench 2.1 under Terminus2-XML, a gain of 11.9 percentage points. These scores average six evaluation runs.
- 03
Starting from 17,558 records at 53.2% on Terminal-Bench 2.1, doubling the corpus through additional environments reaches 56.0%. Doubling queries per environment reaches 53.8%, while doubling solutions per query reaches 53.9%. All three expanded corpora contain about 35,100 records, making environment diversity the most promising axis in this comparison.
- 04
Cross-workspace tasks pair a writable target project with a read-only reference that implements a missing capability. They are harder for the teacher: pass@1 falls from 72.3% on single-workspace tasks to 49.2%. Adding the verified cross-workspace data to single-workspace SFT raises Terminal-Bench 2.1 from 56.4% to 58.4%.
- 05
Multi-round synthesis preserves workspace state, tracks changing requirements, and turns hidden verifier failures into user feedback. Of the retained sessions, 69.6% include a failure followed by recovery. Adding this data to single-workspace SFT raises EvoCode-Bench v2 MT@4 from 18.4% to 21.0%; the full mixture scores 20.1%, versus 6.3% for the base model. MT@4 credits consecutive successful rounds reached across four runs, stopping credit within each run at its first failure.
Evidence & limits
The training experiments use supervised fine-tuning. They do not establish gains from online RL, resistance to reward hacking under sustained policy optimization, or a general RL scaling law.
The expansion ablation matches record counts, rather than tokens, generation cost, or total compute. Run-to-run variation is substantial, so the observed ordering supports further study rather than a universal budget-allocation rule.
Reconstruction is lossy, and the sufficiency judge checks available project context rather than build success or fidelity to the original repository. Standard Ubuntu containers and generated completion files can miss specialized dependencies or unseen project details. Python accounts for 84.7% of the reconstructed terminal pool's primary languages, limiting distributional coverage.
One teacher model generates tasks, solutions, and verifiers. Although verifier agents do not see the solutions and tests are checked against the initial workspace, shared model blind spots can still admit incorrect solutions. The multi-round evaluation contains only 26 tasks, and adding every data component does not maximize every benchmark score.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen3.5-27B, Qwen3.7-Max
- Algorithms
- Supervised fine-tuning, Trajectory-based environment reconstruction, Cross-workspace task synthesis, Verifier-guided multi-round synthesis
- Benchmarks
- Terminal-Bench 2.0, Terminal-Bench 2.1, EvoCode-Bench v2
Topics