Switchback experiments — when to use?
hardAnswer
- Assign entire population to A or B alternately over time windows (e.g., every hour).
- Analyze paired periods.
- Handles: (1) network effects (whole market same variant).
- (2) marketplace balance.
- Downside: (1) confounded with time-of-day / weekday.
- (2) carryover: treatment period effects bleed into next control period (mitigation: exclude first N minutes).
- Uber uses for driver-side experiments.
- Careful analysis with mixed effects for time.
Check yourself — multiple choice
- Random
- Entire population toggles A/B alternately by time window; handles network + marketplace effects; downside: time confounding + carryover (exclude first N min); Uber driver-side
- Just A/B
- Not real
Switchback: whole pop toggles; handles marketplace; watch time confound.
#experimentation
Practise MLOps & Data Quality
215 interview questions in this topic.