EasyDeepLearn

What is RLHF and how does it fit into training an LLM?

hard

Answer

  • RLHF (Reinforcement Learning from Human Feedback) fine-tunes a pretrained language model to align with human preferences.
  • Steps: (1) supervised fine-tuning on curated demonstrations; (2) train a reward model on human preference pairs; (3) optimize the LLM policy against the reward model using PPO, with a KL penalty to stay close to the reference policy.
  • Recent alternatives: DPO removes the RL step by optimizing preferences directly.
Check yourself — multiple choice
  • RLHF uses PPO to optimize against a learned reward model
  • RLHF replaces pretraining with RL only
  • RLHF requires no human labels
  • DPO uses PPO internally

RLHF  =  SFT  →  reward\mathrm{RLHF}\; = \;\mathrm{SFT}\; \to \;\mathrm{reward} model on preferences  →  PPO\mathrm{preferences}\; \to \;\mathrm{PPO} against the reward model.

#llm#alignment

Practise Reinforcement Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions