How do you balance helpfulness vs harmlessness in RLHF?
hardAnswer
- (1) Multi-headed RM: train two reward heads — one for helpfulness, one for harmlessness — and use a weighted combination during PPO.
- Weights explicitly encode the trade-off.
- (2) Constitutional AI approach: separate 'harmless' and 'helpful' phases of preference generation.
- (3) Rejection sampling with a safety filter — sample many completions, filter unsafe ones, then pick the most helpful.
- Anthropic's HH-RLHF paper (Bai 2022) showed multi-head RM is more stable than a single-head one.
Check yourself — multiple choice
- One reward covers everything
- Multi-head RM (helpful + harmless) with explicit weights, or CAI-style separate phases, or filter+rank sampling
- Impossible to balance
- Skip alignment entirely
Multi-objective RLHF: separate RM heads for helpful + harmless with explicit trade-off weights.
#rlhf#alignment#safety
Practise LLMs & GenAI
214 interview questions in this topic.