EasyDeepLearn

Why do ratio metrics need the delta method?

hard

Answer

  • Ratios like CTR = clicks / impressions have per-user numerators AND denominators → sample-averaged ratio has non-trivial variance.
  • Naive user-level SE is wrong (ignores denominator variability).
  • Delta method gives: Var⁡(N/D)\operatorname{Var}(N / D) ≈ (μN/μD)2  ⋅  [Var⁡(N)/μN2  +  Var⁡(D)/μD2  −  2⋅Cov⁡(N,D)/(μN⋅μD)]  /  n({\mu}_{N} / {\mu}_{D})^{2}\; \cdot \;[\operatorname{Var}(N) / {\mu}_{N}^{2}\; + \;\operatorname{Var}(D) / {\mu}_{D}^{2}\; - \;2 \cdot \operatorname{Cov}(N, D) / ({\mu}_{N} \cdot {\mu}_{D})]\; / \;n.
  • Alternative: bootstrap on user level.
  • Every serious A/B platform (Statsig, Eppo, Facebook's own) uses this for CTR / click-through / like-through.
Check yourself — multiple choice
  • No correction needed
  • Ratio N̄/D̄ has variance depending on both numerator + denominator → delta method or user-level bootstrap; naive SE wrong
  • Same as t-test
  • Random

Ratio metrics: delta method or bootstrap; naive SE is wrong.

#ab-testing#confidence-interval

Practise Statistics Fundamentals

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions