Difference-in-differences for ML experiments — when?

hard

Answer

$When randomization impossible (e.g., can't ethically deny users, city-wide rollout).$
$\mathrm{DiD}\; = \;(Y_{\mathrm{treat}}\mathrm{post}\; - \;Y_{\mathrm{treat}}\mathrm{pre})\; - \;(Y_{\mathrm{ctrl}}\mathrm{post}\; - \;Y_{\mathrm{ctrl}}\mathrm{pre})$
$Assumes parallel trends absent treatment.$
$Use when: (1) staggered rollouts across geographies.$
$(2) policy changes affecting group.$
$(3) natural experiments.$
$Modern: synthetic control constructs weighted combination of control units.$

Check yourself — multiple choice

Same as A/B
$(Y_{\mathrm{treat}}\mathrm{post}\; - \;Y_{\mathrm{treat}}\mathrm{pre})\; - \;(Y_{\mathrm{ctrl}}\mathrm{post}\; - \;Y_{\mathrm{ctrl}}\mathrm{pre})$ ; parallel trends assumption; use when random impossible (staggered rollout / policy / natural); synthetic control extension
Random
Not real

DiD: (T post - T pre) - (C post - C pre); parallel trends.

#experimentation

215 interview questions in this topic.