How do you monitor fairness in production?

hard

Answer

$(1) Track prediction / performance metrics per protected group (gender, race, age, geography).$
$(2) Compute fairness metrics: demographic parity, equal opportunity, equalized odds, calibration by group.$
$(3) Alert when disparity exceeds threshold.$
$(4) Regular audits + reporting.$
$(5) Log inputs + protected attributes for post-hoc analysis.$
$(6) Store predictions with proxies (zip code, name) for indirect discrimination detection.$
$Tools: Fairlearn, AIF360, What-If Tool.$

Check yourself — multiple choice

Not needed
Per-group metrics + demographic parity/equal opp/equalized odds/calibration + alerts + audits + proxy features; Fairlearn / AIF360 / What-If
Random
Only accuracy

Fairness monitoring: per-group metrics + fairness definitions + audits.

#monitoring#safety

215 interview questions in this topic.