How do you monitor prediction uncertainty in production?
hardAnswer
- (1) Log softmax entropy / max prob per prediction.
- (2) MC dropout / ensemble variance for Bayesian models.
- (3) Aggregate: histogram of uncertainty over time.
- (4) Alert if mean uncertainty spikes → OOD input distribution.
- (5) Route high-uncertainty predictions to human review or fallback.
- (6) Selective prediction: refuse when uncertainty > threshold, defer to human.
- Critical for medical / legal / financial ML.
Check yourself — multiple choice
- Random
- Log entropy/max prob per prediction + MC dropout / ensemble variance + histogram over time + alert on mean spike (OOD) + route high-uncertainty to human + selective prediction (defer)
- Just accuracy
- Not real
Uncertainty: entropy + MC + histogram + spike alert + selective.
#monitoring#safety
Practise MLOps & Data Quality
215 interview questions in this topic.