What is interleaving in recsys A/B testing?
hardAnswer
- Alternative to standard A/B for ranking systems.
- Instead of splitting users, merge rankings from both models into one presented list (team-draft or probabilistic interleaving).
- User clicks reveal preference.
- Advantages: (1) 10-100x more sample-efficient than user split.
- (2) Same user compares both.
- (3) Fast decisions.
- Used by Netflix, Bing, Yahoo.
- Downside: only measures ranking preference, not business KPI at page level.
Check yourself — multiple choice
- Random
- Merge rankings from both models into one list (team-draft/probabilistic interleave); clicks reveal preference; 10-100x sample-efficient vs user split; Netflix/Bing/Yahoo
- Same as A/B
- Not real
Interleaving: merge rankings, click reveals preference; 10-100x more efficient.
#experimentation
Practise MLOps & Data Quality
215 interview questions in this topic.