Constitutional AI — mechanism.
hardAnswer
- Anthropic 2022.
- Two phases: (1) SL from AI feedback: model critiques + revises its own responses per written constitution (helpful, harmless principles); train on (prompt, revised response).
- (2) RL from AI feedback: LLM judge scores pairs of responses on constitutional principles; train reward model + PPO.
- Advantage: avoids exposing human labelers to harmful content.
- Foundation of Claude's training.
- Modern LLM alignment default.
Check yourself — multiple choice
- Random
- Two phases: (1) LLM self-critiques + revises per constitution → SL; (2) LLM judge scores pairs → RM + PPO; avoids harmful content exposure to human labelers; Claude foundation
- Same as RLHF
- Not real
Constitutional AI: self-critique + LLM judge; foundation of Claude alignment.
#llm#alignment#safety
Practise Reinforcement Learning
214 interview questions in this topic.