How does Bayesian A/B testing work?

medium

Answer

$p_{A}$
$P(p_{B}\; > \;p_{A}\; \mid \;\mathrm{data})$
$(p_{B}\; - \;p_{A})$
$Advantages: interpretable ('72% chance B is better') and safe under peeking (no p-hacking inflation, decisions optimal under a loss function).$
$Standard in growth teams alongside frequentist.$

Check yourself — multiple choice

Same as p-value
Beta-Bernoulli conjugate posteriors → $P(p_{B}\; > \;p_{A}\; \mid \;\mathrm{data})$ via Monte Carlo; peek-safe; direct interpretation as probability of winning
Not real
Only frequentist

Bayesian A/B: Beta $\mathrm{posteriors}\; + \;P(p_{B}\; > \;p_{A}\; \mid \;\mathrm{data})$ ; peek-safe.

#bayesian#ab-testing

215 interview questions in this topic.