Safety verifier reward — what and how?
hardAnswer
- Automatic reward from external verifier that judges safety: (1) content filter classifier flags harmful outputs.
- (2) LLM judge with harm rubric.
- (3) Rule-based checks (PII leakage, illegal content).
- Combined with helpfulness reward → multi-objective RL.
- Alignment tax risk if safety over-weighted.
- Modern: 'safety as red-team' — actively probe model with attacks, RL trains against them (Anthropic's HH pipeline).
Check yourself — multiple choice
- Random
- Automatic reward from harm classifier / LLM judge / rule-based checks; combined with helpfulness → multi-objective; risk of alignment tax; 'safety as red-team' probes model with attacks
- Same as filter
- Not real
Safety verifier: harm classifier / LLM judge / rules; multi-obj RL + red-team.
#llm#alignment#safety
Practise Reinforcement Learning
214 interview questions in this topic.