Glossary
RLHF — Glossary
Reinforcement Learning from Human Feedback: aligning models using preference data and a reward model.
Full definition
RLHF typically follows supervised instruction tuning. Humans (or AI raters) compare candidate answers; a reward model learns those preferences; the policy is optimized to score well under that reward while staying close to a reference model.
The InstructGPT paper (Ouyang et al.) popularized this stack for helpful assistants. Variants such as direct preference optimization change the optimizer while pursuing similar goals.
RLHF can reduce some harmful behaviors and improve instruction following, but it can also cause over-refusal or reward hacking if evaluations are narrow.