What is KTO and when is it useful?
hardAnswer
- KTO (Kahneman-Tversky Optimization; Ethayarajh 2024) is a DPO variant that uses only binary 'good/bad' feedback per example (not paired preferences) and asymmetric utility weights (losses hurt more than gains — Kahneman-Tversky prospect theory).
- Practically useful when you have thumbs-up / thumbs-down user feedback (much more common than paired A/B preference) — no need to construct chosen/rejected pairs.
- Matches DPO quality on many benchmarks.
Check yourself — multiple choice
- KTO requires paired preferences
- Alignment from unpaired binary good/bad feedback with asymmetric utility — usable directly from thumbs-up/down
- Same as PPO
- Only for math
KTO: binary good/bad feedback, no pairing needed → practical for real-world user thumbs data.
#alignment#preference
Practise LLMs & GenAI
214 interview questions in this topic.