How is 'refusal' behavior trained into LLMs?
mediumAnswer
- SFT data explicitly includes examples where the user asks for something harmful / disallowed and the assistant refuses with a helpful explanation.
- Preference data marks 'good refusals' over 'harmful compliance' and over 'over-refusal'.
- RLHF/DPO stages then further shape refusal.
- Trade-off: too much refusal training → over-refusal ('I can't help with that' for benign queries); too little → jailbreaks.
- Frontier labs iterate this trade-off constantly.
Check yourself — multiple choice
- Refusals happen randomly
- SFT + preference data with harmful/benign examples; balance under- vs over-refusal
- Only regex filters
- Never trained
Refusal behavior: engineered via SFT + preference data; balance harm vs over-refusal.
#alignment#safety
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What is Constitutional AI?
- How do you balance helpfulness vs harmlessness in RLHF?
- What is an instruction hierarchy in modern LLMs?
- What are the main LLM safety and alignment concerns?
- What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?
- How is high-quality instruction-tuning data constructed?