Long-term holdout — what and why?

hard

Answer

$Reserve 1-10% of users as permanent control who never receive new model / experiment.$
$Measure long-term effects of everything shipped combined.$
$Detect novelty / primacy effects, learned tolerances, and cumulative regressions that individual A/B tests miss.$
$Standard at Google / Facebook / Netflix.$
$Rotate holdout every 6-12 months.$
$Small ongoing revenue cost, large diagnostic benefit.$

Check yourself — multiple choice

Random
Reserve 1-10% users as permanent control (never receive experiments); measure long-term cumulative effects + novelty/primacy + tolerance; Google/FB/Netflix; rotate every 6-12 months
Same as test
Not real

Long-term holdout: permanent control; measure cumulative + novelty effects.

#experimentation

215 interview questions in this topic.