TL;DR
A randomly initialized program generator learns through RL to produce synthetic byte sequences that train a separate learner. The resulting small models improve zero-shot prediction across natural-data modalities as self-play compute increases, without using natural data for gradient updates.
Why it matters
This is a controlled test of whether compute can produce transferable training structure without drawing new examples from a natural corpus. It separates an adaptive synthetic curriculum from fixed program sampling and tests transfer beyond the generated data.
Key findings
- 01
Two independently parameterized Llama-style transformers co-evolve. The generator emits executable programs in a Brainf*ck-based language; program outputs train the learner with next-byte cross-entropy. RL rewards the generator using a learning-progress signal based on alignment with the learner’s recent gradient trajectory.
- 02
The experiments use a 256-byte vocabulary, a 4,096-token context, and models below 25M parameters. Compute-optimal frontiers are constructed across model sizes, checkpoints, and ensembles, with held-out loss measured in bits per byte.
- 03
Self-play transfers across text, images, music, audio, speech, and other byte-encoded domains and scales faster than fixed sampling from the same universal program space. A hand-designed PCFG prior remains stronger on text and code, while self-play transfers better to several non-language modalities.
- 04
In a separate 24.4M-parameter warm-start experiment, self-play initialization reduces natural-data tokens consumed at convergence from 496M to 320M on ESC-50 audio and from 588M to 421M on CIFAR-10. These downstream comparisons exclude the preceding self-play compute.
Evidence & limits
“Zero data” means no natural data for gradient updates during self-play. Hyperparameters are selected using validation loss on DCLM and DNA; the paper explicitly acknowledges limited leakage through this selection.
The experiments remain below 25M parameters with a 4K context. Power-law fits over measured compute frontiers do not establish frontier-model capabilities or unlimited extrapolation.
Downstream token savings omit the cost of self-play pre-pretraining. They are not evidence of lower total compute for a single downstream run.
Natural-data prediction loss and synthetic in-context-learning tasks do not demonstrate factual knowledge or general assistant ability. The authors explicitly frame self-play as complementary to natural data, which supplies world-specific information.
The program space includes selected macro instructions, and observed mathematical sequences are not independently established as the cause of transfer. The result is a proof of concept with architectural and curriculum choices.
Scaling dimensions
Models, methods & benchmarks
- Models
- Llama-style transformers below 25M
- Algorithms
- Self-play, Learning-progress rewards, Autoregressive pretraining
- Benchmarks
- DCLM, CIFAR-10, ESC-50, DNA, Mutopia, Metamath
Topics