Direct Preference Optimization: Your Language Model is Secretly a Reward Model

TL;DR

DPO reformulates preference optimization as a direct classification-style objective, avoiding an explicit reward-model training stage and online policy optimization.

Why it matters

It established a simpler baseline for preference post-training and sharpened the question of when the complexity of online RL is actually necessary.

Key findings

1. 01

   A closed-form relationship connects the optimal policy and reward model under the KL-constrained objective.

2. 02

   Direct optimization reduces pipeline complexity for many preference-learning settings.

Scaling dimensions

Models, methods & benchmarks

Algorithms

DPO

Topics