EasyDeepLearn

KL divergence — what it measures and pitfalls.

medium

Answer

  • KL⁡(P  ∣∣  Q)  =  Σ\operatorname{KL}(P\; \mid \mid \;Q)\; = \;{\Sigma} P(x) log⁡(P(x)/Q(x))\operatorname{log}(P(x) / Q(x)) = -H(P) + H(P, Q).
  • Not symmetric, not a metric.
  • Zero iff P = Q, infinite if Q(x) = 0 where P(x) > 0.
  • Two directions: forward KL⁡(P∣∣Q)\operatorname{KL}(P \mid \mid Q) 'mode-covering' (VAE default → q spreads over p); reverse KL⁡(Q∣∣P)\operatorname{KL}(Q \mid \mid P) 'mode-seeking' (VI → q concentrates on one mode of p).
  • Standard divergence in ML but rarely a metric; prefer Wasserstein / MMD when metric properties matter.
Check yourself — multiple choice
  • Symmetric metric
  • KL⁡(P∣∣Q)  =  Σ\operatorname{KL}(P \mid \mid Q)\; = \;{\Sigma} P log⁡(P/Q)\operatorname{log}(P / Q); not symmetric / not a metric; forward mode-covering (VAE), reverse mode-seeking (VI); prefer Wasserstein/MMD when metric needed
  • Random
  • Not useful

KL: forward mode-covering, reverse mode-seeking; not symmetric / metric.

#density#theory

Practise Unsupervised Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions