Direct preference optimisation
Aligning a model to human preferences without an explicit reward model, by optimising on chosen-versus-rejected pairs.
By Lattice editorial · Entry last reviewed · method
What it implies
Removes the reward-model training step and its instabilities, which is why it largely replaced RLHF in practice. It needs preference data, which is expensive to produce well — and is where most projects quietly stall.
Tools in this index
Related terms
- Supervised fine-tuningTraining on curated input/output pairs so the model adopts a specific task or response style.
- RLHFReinforcement learning from human feedback: training a reward model from preferences, then optimising the policy against it.
- GRPOGroup relative policy optimisation — a reinforcement-learning method that compares a group of sampled answers rather than learning a separate reward model.