What are guardrail metrics in an A/B test?

medium

Answer

$Metrics you don't want to regress even if primary metric improves.$
$Examples: (1) latency p99 (should not degrade).$
$(2) error rate.$
$(3) revenue (never negative).$
$(4) safety violations (harmful content).$
$(5) fairness metrics per group.$
$(6) user complaints / bounce rate.$
$Set two-sided bounds: primary must be significantly UP, guardrails must NOT be significantly DOWN.$
$Pre-registered before test.$
$Non-inferiority testing framework.$

Check yourself — multiple choice

Only primary
Metrics that must NOT regress: latency / error / revenue / safety / fairness / bounce; two-sided (primary up + guardrails not down); pre-registered; non-inferiority framework
Random
Not real

Guardrails: must-not-regress metrics; pre-registered; non-inferiority.

#experimentation

Practise MLOps & Data Quality

215 interview questions in this topic.

Read the lesson Flashcards Take the quiz

Related questions