A/B test vs multi-armed bandit — which to use?

hard

Answer

$A/B: fixed allocation, wait for significance, use when you need clean causal comparison + all variants matter equally.$
$Bandit (Thompson sampling / UCB): dynamically allocate more to winning variants — minimizes regret.$
$Use when (1) many variants (10+), (2) speed matters over precision, (3) revenue impact during test matters.$
$Downside: unbalanced sample \to weaker inference.$
$Modern: contextual bandit for personalized allocation.$
$Netflix, Google use both.$

Check yourself — multiple choice

Random
A/B: fixed allocation + clean causal; bandit: dynamic allocation minimizes regret; A/B for hypothesis, bandit for many variants + speed + revenue during test; contextual bandit for personalization
Same
Not real

A/B for clean causal; bandit for many variants + revenue during test.

#experimentation

215 interview questions in this topic.