Training Language Models to Follow Instructions with Human Feedback

TL;DR

The InstructGPT work combines demonstrations, preference comparisons, reward modeling, and PPO to align language-model behavior with user instructions.

Why it matters

It established the modern RLHF pipeline as a practical post-training system and made data collection, reward modeling, and policy optimization a shared engineering problem.

Key findings

1. 01

   Post-training can materially change user-perceived behavior without simply increasing model size.

2. 02

   The quality of demonstrations and comparisons is central to the result.

Scaling dimensions

Models, methods & benchmarks

Models

InstructGPT

Algorithms

PPO, RLHF

Topics