OEC — Overall Evaluation Criterion — what is it?

hard

Answer

$A single scalar composite of relevant metrics chosen ex ante as the experiment's success criterion.$
$Simplifies decision-making (one number).$
$Should reflect business value, be sensitive at reasonable sample sizes, and align long-term with company OKRs.$
$Examples: Bing's 'sessions per user', Netflix engagement composite.$
$Cost: hard to design well; teams often mis-weight components \to dumb decisions.$
$Modern practice: use OEC + a small guardrail suite rather than OEC alone.$

Check yourself — multiple choice

Random
Single scalar composite metric chosen ex ante; simplifies decision-making but must reflect long-term business value; pair with guardrails
Same as p-value
Not real

OEC: single composite success metric; simplify decisions + guardrails.

#ab-testing#design

215 interview questions in this topic.