Deep Reinforcement Learning from Human Preferences

TL;DR

A foundational study of learning a reward model from pairwise human preferences and using that learned reward to train an RL agent.

Why it matters

The paper defines the central scalable-supervision loop behind later RLHF systems: collect comparisons, fit a reward proxy, optimize behavior, and repeat.

Key findings

1. 01

   Pairwise preferences can supervise behavior when hand-written rewards are difficult.

2. 02

   Reward learning and policy learning form an iterative data flywheel.

Scaling dimensions

Models, methods & benchmarks

Algorithms

Preference-based RL

Topics