When do you need calibrated probabilities rather than a good ranking?
mediumAnswer
- You need calibration whenever the probability itself enters a downstream calculation.
- Expected-value decisions, such as multiplying a predicted default probability by a loan amount, break if the number is not a real probability.
- Blending several model outputs, setting a fixed cost threshold, or reporting risk to a regulator all need calibration.
- A pure ranking is enough when you only take the top k, since a monotone transform does not change the order.
- Note that AUC is invariant to calibration, so a model can have excellent AUC and wildly overconfident probabilities.
Check yourself — multiple choice
- Never, ranking is always enough
- When the probability feeds an expected-value calculation, a fixed cost threshold, model blending or reporting — AUC cannot detect miscalibration
- Only for neural networks
- Only when accuracy is low
Calibration matters when the probability value is used, not just its order; AUC is blind to it.
#calibration#metrics
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is log loss (cross-entropy) and what does a specific value mean?
- What is the Brier score and how does it compare to log loss?
- How do you check whether a classifier's probabilities are well-calibrated?
- What is Youden's J statistic and how is it used?
- How do you select the decision threshold for a binary classifier?
- Precision vs recall — when do you optimize each?