You must pick ONE metric for your model. How do you decide?
mediumAnswer
- Start from the business objective, not the model: what does 'error' cost, and to whom?
- Classification with asymmetric costs → expected cost or F-beta with the appropriate beta.
- Rare-positive detection → PR-AUC / recall at fixed precision.
- Ranking → NDCG or MAP.
- Probabilistic forecasts → log loss or Brier + calibration.
- Regression with tail-critical decisions → quantile loss.
- Regression with symmetric costs and Gaussian errors → RMSE.
- Always report multiple metrics; the *primary* one is the one you optimize and stop-arg on.
Check yourself — multiple choice
- Always use accuracy
- Match the metric to the downstream cost / decision — cost, ranking, calibration, tail-risk, etc.
- Always the metric your model minimizes during training
- Whatever your competitor uses
Pick the metric from the downstream cost / decision, not from the model.
#metrics#evaluation#interview-scenarios
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- What is the bias-variance tradeoff?
- How do you detect and fix overfitting?
- Precision vs recall — when do you optimize each?
- When should you use PR-AUC instead of ROC-AUC?
- What is k-fold cross-validation and when do you use stratified or grouped folds?
- How do you read a confusion matrix and derive the key metrics?