What is log-cosh loss and its advantage over Huber?
mediumAnswer
- L(y, ŷ) = log(cosh(y - ŷ)) ≈ for small errors -ŷ| - for large — same behavior as Huber but smooth everywhere (all derivatives exist).
- No δ hyperparameter.
- Slightly more expensive to compute.
- Use it when you want Huber's outlier-robustness without tuning δ, and gradient smoothness matters (e.g., second-order optimizers).
How to say it out loud
Log-cosh is the log of the hyperbolic cosine of the residual. For small errors it behaves like squared error, for large errors it grows linearly, so it is robust to outliers in the same way Huber is. The difference is that it is smooth everywhere, including across the transition, and there is no delta to tune. Huber has a kink at delta where the second derivative does not exist, which matters the moment you use anything second-order. What you give up is a slightly more expensive forward pass and any explicit control over where the transition sits — with Huber you choose it, with log-cosh you take what the function gives you.
Check yourself — multiple choice
- Discontinuous
- — Huber-like behavior but smooth everywhere and no hyperparameter
- Same as MAE
- Only for classification
log-cosh: smooth everywhere Huber-alternative, no δ.
What the interviewer asks next
- Write the Huber loss and explain when to use it.You cannot argue log-cosh is the better choice without putting Huber on the board first.
- MSE vs MAE for regression — when is each preferred?Both losses exist to interpolate between these two, so this is the question underneath the question.
- How should the output head be designed for a regression task with a strictly positive target?The loss is only half of a regression answer; the head is the half candidates forget.
#losses
Practise Deep Learning
214 interview questions in this topic.