EasyDeepLearn

What is an A/A test and why run it?

medium

Answer

  • Both variants receive the SAME model.
  • Expected result: no significant difference (~5% false-positive rate).
  • Uses: (1) validate experimentation platform (SRM, randomization correctness).
  • (2) baseline noise level for the metric.
  • (3) sanity check before running A/B (ensure no accidental config difference).
  • Run periodically.
  • Standard practice at Airbnb, Booking, Microsoft.
  • Failure = broken platform, don't trust any results until fixed.
Check yourself — multiple choice
  • Random
  • Both variants = same model; expect no significant diff (~5% FP); validates platform + baseline noise + sanity check before A/B; run periodically; failure = broken platform
  • Same as A/B
  • Not real

A/A test: sanity-check platform; expect no diff; run periodically.

#experimentation

Practise MLOps & Data Quality

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions