EasyDeepLearn
Supervised Learning · section 10 of 18

Regression metrics

6 interview questions on regression metrics, each answered in full. Free to read, no account needed.

RMSE vs MAE — how do you pick?

easy
  • RMSE = sqrt(mean((y  −  yhat)2)\mathrm{mean}((y\; - \;y_{\mathrm{hat}})2)) — same units as y, penalizes large errors quadratically.
  • Use when big errors are disproportionately bad (financial forecasting, energy grids) or when errors are approximately Gaussian.
  • MAE⁡  =  mean(∣y  −  yhat∣)\operatorname{MAE}\; = \;\mathrm{mean}( \mid y\; - \;y_{\mathrm{hat}} \mid ) — robust to outliers, more interpretable ('average error is X units'), the L1 relative of RMSE.
  • RMSE ≥ MAE always; the gap grows with error variance.
  • If a few large errors would dominate, prefer MAE (or Huber).
#metrics-regressionPermalink & quiz →

What are the pitfalls of MAPE (Mean Absolute Percentage Error)?

medium
  • MAPE  =  mean(∣y  −  yhat∣  /  ∣y∣)  ⋅  100\mathrm{MAPE}\; = \;\mathrm{mean}( \mid y\; - \;y_{\mathrm{hat}} \mid \; / \; \mid y \mid )\; \cdot \;100%.
  • Issues: (1) undefined when y = 0 and explodes when y is close to 0; (2) asymmetric — over-forecasting is bounded (max 100% error), under-forecasting is unbounded; (3) biases model selection toward under-prediction.
  • Alternatives: sMAPE (symmetric), MASE (compare to naive forecast), WAPE (weighted / total absolute error / total absolute y), or use quantile losses.
  • Never use MAPE with targets that can be zero or negative.
#metrics-regressionPermalink & quiz →

What is sMAPE and why is it used?

medium
  • Symmetric MAPE = mean(|y  −  yhaty\; - \;y_{\mathrm{hat}}| / ((∣y∣  +  ∣yhat∣)/2)(( \mid y \mid \; + \; \mid y_{\mathrm{hat}} \mid ) / 2)) * 100%.
  • Bounded in [0%, 200%] and symmetric in over-/under-prediction: over-forecasting and under-forecasting by the same absolute amount give equal error.
  • Still undefined when both y and yhaty_{\mathrm{hat}} are 0.
  • Better than MAPE for time series forecasting competitions (used in M-competitions).
  • Not a proper scoring rule for probabilistic forecasts — MASE (mean absolute scaled error) is a better alternative in many cases.
#metrics-regressionPermalink & quiz →

What is explained variance score and how does it differ from R2R^{2}?

hard
  • explainedvariance  =  1  −  Var⁡(y  −  yhat)  /  Var⁡(y)\mathrm{explained}_{\mathrm{variance}}\; = \;1\; - \;\operatorname{Var}(y\; - \;y_{\mathrm{hat}})\; / \;\operatorname{Var}(y).
  • It measures the fraction of variance the model captures.
  • R2  =  1  −  SSres  /  SStotR^{2}\; = \;1\; - \;\mathrm{SS}_{\mathrm{res}}\; / \;\mathrm{SS}_{\mathrm{tot}} uses the sum of squared residuals directly.
  • They are equal *only when the residuals have zero mean* (i.e., predictions are unbiased on average).
  • If predictions are systematically biased, explained variance can be high while R2R^{2} is much lower.
  • Use R2R^{2} for model comparison; use explained variance to diagnose bias-vs-variance in the errors.
#metrics-regressionPermalink & quiz →

When would you use MSLE (mean squared log error) instead of MSE?

hard
  • MSLE = mean((log⁡(1+y)  −  log⁡(1+yhat))2(\operatorname{log}(1 + y)\; - \;\operatorname{log}(1 + y_{\mathrm{hat}}))2).
  • Penalizes relative errors instead of absolute ones — an error of 10 units when y=1000 is treated the same as an error of 1 unit when y=100.
  • Use when the target spans many orders of magnitude (population sizes, incomes, energy consumption).
  • Requires y ≥ 0.
  • Also under-penalizes over-prediction less than under-prediction — biased slightly toward higher predictions.
  • RMSLE (square root) is more interpretable.
#metrics-regressionPermalink & quiz →

Is R2R^{2} a useful metric for non-linear models (RF, GBM, neural nets)?

medium
  • It's a valid summary but has caveats: (1) R2R^{2} is not scale-free per dataset — it depends on Var⁡(y)\operatorname{Var}(y) — so comparing across datasets is misleading; (2) it's easy to inflate by adding features and can overstate goodness for wiggly models on small data; (3) it doesn't reflect calibration or heteroscedastic errors.
  • Prefer RMSE / MAE / quantile loss for absolute quality, MASE for time series comparison, and Bayesian criteria (AIC, BIC) for model selection.
#metrics-regressionPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz