EasyDeepLearn
Supervised Learning · section 9 of 18

Classification metrics

6 interview questions on classification metrics, each answered in full. Free to read, no account needed.

When should you use PR-AUC instead of ROC-AUC?

medium
  • Use PR-AUC when the positive class is rare (heavy imbalance).
  • ROC-AUC can look optimistic on imbalanced data because the true-negative rate dominates the false-positive rate.
  • PR-AUC only involves the positive class (precision and recall), so it reflects real performance on the minority class.
#metrics#imbalancePermalink & quiz →

How do you check whether a classifier's probabilities are well-calibrated?

medium
  • Draw a reliability diagram: bin predictions by predicted probability (e.g., 10 bins), for each bin plot mean predicted probability (x) against fraction of positives (y).
  • A perfectly calibrated model lies on the diagonal.
  • Also report ECE (Expected Calibration Error), the average absolute gap between predicted and empirical frequency across bins.
  • Alternatives: quantile bins for imbalanced data, log-loss / Brier as summary metrics.
#calibration#metricsPermalink & quiz →

Platt scaling vs isotonic regression for probability calibration — how do you choose?

medium
  • Platt: fit a logistic regression on the classifier's scores → sigmoid recalibration.
  • Parametric (2 params), works well with small calibration sets, assumes a sigmoidal miscalibration shape.
  • Isotonic: fit a non-decreasing step function via pool-adjacent-violators.
  • Non-parametric, more flexible, but needs more data (~1000+ examples) or it overfits.
  • Rule of thumb: Platt for small calibration sets or nearly-calibrated models; isotonic for larger data or arbitrary miscalibration curves.
#calibrationPermalink & quiz →

How do you choose the decision threshold for a classifier in a business setting?

medium
  • Not at 0.5.
  • Write down the cost of a false positive and the value of a true positive, then pick the threshold that maximizes expected value.
  • If a false negative costs 10 times a false positive, the optimal threshold moves well below 0.5.
  • When costs are unknown, work backwards from a capacity constraint: if the team can only call 500 customers a week, take the top 500 scores and report the precision you get there.
  • Always choose the threshold on a validation set and confirm on a held-out set, because the optimum shifts with the base rate.
#metrics#metrics-classificationPermalink & quiz →

When do you need calibrated probabilities rather than a good ranking?

medium
  • You need calibration whenever the probability itself enters a downstream calculation.
  • Expected-value decisions, such as multiplying a predicted default probability by a loan amount, break if the number is not a real probability.
  • Blending several model outputs, setting a fixed cost threshold, or reporting risk to a regulator all need calibration.
  • A pure ranking is enough when you only take the top k, since a monotone transform does not change the order.
  • Note that AUC is invariant to calibration, so a model can have excellent AUC and wildly overconfident probabilities.
#calibration#metricsPermalink & quiz →

Your positive class is 0.5%. Is resampling your first move?

medium
  • Usually not.
  • First check whether you have an evaluation problem rather than a training problem: switch from accuracy to precision-recall AUC or precision at the operating point, since accuracy is meaningless at that base rate.
  • Then try class weights, which are cheaper than resampling and leave the data distribution alone.
  • Boosting with scaleposweight\mathrm{scale}_{\mathrm{pos}}\mathrm{weight} often handles 1:200 without any resampling.
  • Reach for SMOTE last, and only inside the cross-validation folds, because oversampling before splitting leaks synthetic neighbours across the split.
  • Remember resampling distorts predicted probabilities and needs recalibration afterwards.
#imbalance#metrics-classificationPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz