Learning to Summarize from Human Feedback

TL;DR

A large-scale study of training summarization models from human comparisons using reward modeling and reinforcement learning.

Why it matters

It is an early, clear demonstration that preference data can optimize qualities that are difficult to specify with a fixed automatic metric.

Key findings

1. 01

   Human comparisons provide a trainable signal for subjective output quality.

2. 02

   Reward-model generalization becomes a central constraint as policy optimization increases.

Scaling dimensions

Models, methods & benchmarks

Algorithms

RLHF, PPO

Topics