When should you use a bandit instead of an A/B test?
hardAnswer
- Bandit (Thompson sampling / UCB) automatically shifts traffic toward the winning arm during the test → minimizes regret.
- Use when: (1) short-lived items (news, promotions) where you can't afford to send 50% traffic to the loser; (2) many arms (>10) with fast feedback; (3) time-decaying decisions.
- Don't use when: (1) you need clean inference on effect size / statistical significance for stakeholder communication; (2) delayed feedback; (3) SUTVA-violating settings.
- Bandits are for optimization, A/B is for learning.
Check yourself — multiple choice
- Always bandit
- Bandits: short-lived / many-arm / fast feedback → minimize regret; A/B: clean inference + effect sizes for stakeholder decisions → learning
- Same thing
- Random
Bandits: for optimization; A/B: for learning / inference.
#ab-testing#bayesian
Practise Statistics Fundamentals
215 interview questions in this topic.