Why do offline improvements often not translate online?

hard

Answer

$(1) Distribution shift: offline eval on historical data doesn't reflect current traffic.$
$(2) Feedback effects: recsys changing displayed items changes user behavior.$
$(3) Metric mismatch: offline accuracy vs online business KPI.$
$(4) Latency degradation: model too slow at production scale.$
$(5) Serving-training skew: features computed differently.$
$(6) Novelty effect: users react to change itself, not model quality.$
$(7) Long-tail failures: rare edge cases dominate real users.$
$Always validate offline gains with online A/B.$

Check yourself — multiple choice

Random
Distribution shift / feedback effects / metric mismatch / latency / serve-train skew / novelty / long-tail failures; always validate offline wins with online A/B
Always matches
Not real

Offline≠Online: shift / feedback / metric / latency / skew / novelty / tail.

#deployment#monitoring

215 interview questions in this topic.