Overall conversion went down but improved in every country. How is that possible?

hard

Answer

$The traffic mix changed.$
$Simpson's paradox appears when the groups have different baseline rates and their relative share shifts, so an aggregate can move opposite to every subgroup.$
$Concretely, if a low-converting country grew as a fraction of traffic, the overall rate falls even though each country improved.$
$This is why aggregate metrics over a heterogeneous population are unreliable when composition is not held fixed, which happens constantly with marketing campaigns and regional launches.$
$The remedy is to compare like with like: stratify by the confounding variable and report a weighted average using a fixed reference composition, or model the outcome with the segment included.$
$In a randomized experiment this cannot happen by chance across arms, so if it does, suspect a broken assignment or differential logging.$

Check yourself — multiple choice

It indicates a data error
Simpson's paradox from a shift in traffic composition across groups with different baselines — stratify and reweight to a fixed reference mix, and in a randomized test suspect broken assignment
It is impossible
The subgroup results must be wrong

Changing group shares with differing baselines can reverse the aggregate; fix the composition to compare fairly.

#causal-inference#descriptive

215 interview questions in this topic.