How do you check whether a classifier's probabilities are well-calibrated?
mediumAnswer
- Draw a reliability diagram: bin predictions by predicted probability (e.g., 10 bins), for each bin plot mean predicted probability (x) against fraction of positives (y).
- A perfectly calibrated model lies on the diagonal.
- Also report ECE (Expected Calibration Error), the average absolute gap between predicted and empirical frequency across bins.
- Alternatives: quantile bins for imbalanced data, log-loss / Brier as summary metrics.
Check yourself — multiple choice
- By looking at accuracy only
- Reliability diagram (predicted vs empirical frequency per bin) plus ECE
- By computing on probabilities
- By plotting the ROC curve
Reliability diagram + ECE: how close is to observed frequency, per bin?
#calibration#metrics
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is log loss (cross-entropy) and what does a specific value mean?
- What is the Brier score and how does it compare to log loss?
- What is Youden's J statistic and how is it used?
- How do you select the decision threshold for a binary classifier?
- When do you need calibrated probabilities rather than a good ranking?
- Precision vs recall — when do you optimize each?