EasyDeepLearn

Derive the ELBO for VI.

hard

Answer

  • log p(D) = log ∫ p(D, θ) dθ = log ∫ q(θ) * p(D, θ)/q(θ) dθ ≥ Eq[log⁡  p(D,  θ)/q(θ)]E_{q}[\operatorname{log}\;p(D, \;{\theta}) / q({\theta})] (Jensen).
  • ELBO = E_q[log p(D, θ)] - E_q[log q(θ)] = E_q[log p(D | θ)] - KL(q || p(θ)). log p(D)  −  ELBO  =  KL⁡(q  ∣∣  p(θ  ∣  D))p(D)\; - \;\mathrm{ELBO}\; = \;\operatorname{KL}(q\; \mid \mid \;p({\theta}\; \mid \;D)) ≥ 0 → maximizing ELBO minimizes KL to the posterior.
  • Same objective drives VAE training in deep learning.
Check yourself — multiple choice
  • Random
  • ELBO  =  Eq[log⁡  p(D∣θ)]  −  KL⁡(q  ∣∣  prior)\mathrm{ELBO}\; = \;E_{q}[\operatorname{log}\;p(D \mid {\theta})]\; - \;\operatorname{KL}(q\; \mid \mid \;\mathrm{prior}); log p(D)  −  ELBO  =  KL⁡(q  ∣∣  posterior)p(D)\; - \;\mathrm{ELBO}\; = \;\operatorname{KL}(q\; \mid \mid \;\mathrm{posterior}) ≥ 0 → maximizing ELBO minimizes KL to posterior; VAE loss too
  • Same as log-lik
  • Not real

ELBO  =  Eq[log⁡  p(D∣θ)]  −  KL⁡(q  ∣∣  prior)\mathrm{ELBO}\; = \;E_{q}[\operatorname{log}\;p(D \mid {\theta})]\; - \;\operatorname{KL}(q\; \mid \mid \;\mathrm{prior}); log p(D)  −  ELBO  =  KL⁡(q  ∣∣  posterior)p(D)\; - \;\mathrm{ELBO}\; = \;\operatorname{KL}(q\; \mid \mid \;\mathrm{posterior}).

#bayesian#mcmc

Practise Statistics Fundamentals

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions