What are guardrail metrics?
easyAnswer
- Secondary metrics you commit to monitor to catch bad tradeoffs: latency, error rate, retention, revenue-per-user, complaint rate, security signals.
- Test can 'win' on the primary metric but must not degrade guardrails beyond a preset threshold.
- Standard practice: bidirectional CI on each guardrail must not cross the harm threshold.
- Prevents 'winning' by cannibalizing another team's metric or long-term health.
Check yourself — multiple choice
- Random
- Committed secondary metrics (latency, retention, revenue, errors) that must not degrade beyond threshold — prevents cannibalizing wins
- Only primary
- Not real
Guardrail metrics: monitored secondary metrics; prevent cannibalizing tradeoffs.
#ab-testing#design
Practise Statistics Fundamentals
215 interview questions in this topic.
Related questions
- What are the pillars of a solid A/B test design?
- What is Sample Ratio Mismatch (SRM) and why check it?
- OEC — Overall Evaluation Criterion — what is it?
- How does a company scale to running 1000+ experiments concurrently?
- What is sampling bias and how do you mitigate it?
- What is MDE (minimum detectable effect)?