Reinforced Self-Training (ReST) for Language Modeling

TL;DR

ReST alternates offline sample generation with policy improvement, allowing generated data to be reused rather than coupling every update to fresh online rollouts.

Why it matters

Rollout generation is expensive. Reusing experience can improve sample efficiency and operational simplicity when the offline objective remains well aligned.

Key findings

1. 01

   Separating generation and improvement enables data reuse.

2. 02

   Growing-batch training offers a practical alternative to fully online RLHF loops.

Scaling dimensions

Models, methods & benchmarks

Algorithms

ReST, Offline RL

Topics