What are the main pitfalls of relying on ROC-AUC?
mediumAnswer
- (1) On heavily imbalanced data, ROC-AUC is optimistic because the false positive rate is dominated by a huge number of negatives — PR-AUC is more informative.
- (2) AUC is threshold-independent — a model with great AUC can still have terrible precision at any usable threshold.
- (3) It doesn't reward calibration — two models with identical AUC can have very different probability quality.
- (4) Comparing AUC across datasets with different class balances is misleading.
- Always pair AUC with a calibration metric and a cost-aware operating point.
Check yourself — multiple choice
- AUC is always the best single metric
- AUC is optimistic on imbalance, threshold-independent, and blind to calibration
- AUC handles imbalance perfectly
- AUC replaces the need for a threshold
AUC ≠ calibration, is optimistic on imbalance, and threshold-independent.
#metrics#classification#imbalance
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Why is accuracy a bad primary metric for many real-world problems?
- What is Cohen's kappa and why is it useful?
- What is the Matthews Correlation Coefficient (MCC) and when should you use it?
- What is balanced accuracy?
- What is the geometric mean (G-mean) in classification metrics?
- How do you select the decision threshold for a binary classifier?