EasyDeepLearn

What is Fisher information?

hard

Answer

  • I(θ) = -E[∂2  log⁡  p(X  θ)  /  ∂θ2]  =  E[(∂  log⁡  p(X  θ)/∂θ)2]E[ \partial ^{2}\;\operatorname{log}\;p(X\;{\theta})\; / \; \partial {\theta}^{2}]\; = \;E[( \partial \;\operatorname{log}\;p(X\;{\theta}) / \partial {\theta})^{2}].
  • Measures how much information the observations carry about θ.
  • Cramér-Rao bound: for any unbiased estimator, Var⁡(θ)\operatorname{Var}({\theta}) ≥ 1 / (n * I(θ)).
  • MLE asymptotic variance = 1 / (n * I(θ)) → MLE is asymptotically efficient.
  • Used to define natural gradient (natural  gradient  descent  ∝  I(θ)−1  ∇L)(\mathrm{natural}\;\mathrm{gradient}\;\mathrm{descent}\; \propto \;I({\theta})^{-1}\; \nabla L).
Check yourself — multiple choice
  • Random
  • I(θ) = curvature of log-likelihood; Cramér-Rao: Var⁡(θ)\operatorname{Var}({\theta}) ≥ 1/(n·I); MLE reaches this asymptotically; foundation of natural gradient
  • Same as MSE
  • Not useful

Fisher information: curvature of log-lik; CR lower bound; natural gradient basis.

#estimation#theory

Practise Statistics Fundamentals

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions