RLAIF — RL from AI Feedback.
hardAnswer
- Replace human labelers with LLM judge for preference labels.
- Bai et al. 2022 (Constitutional AI): iteratively use a strong LLM to critique + revise responses per a written constitution.
- Advantages: scale (billions of preferences), consistency, cost.
- Trade-offs: risk of amplifying LLM biases, less alignment with real user preferences.
- Standard in modern alignment (Anthropic's Constitutional AI, Meta's Llama, Google's Gemini all use RLAIF + human).
Check yourself — multiple choice
- Random
- Replace human preference labels with LLM judge (per constitution); scale + consistency + cost; risk of amplifying LLM bias; used in Constitutional AI / Llama / Gemini + human hybrid
- Same as RLHF
- Not real
RLAIF: LLM judge for preferences; Constitutional AI; scales beyond humans.
#llm#alignment
Practise Reinforcement Learning
214 interview questions in this topic.