How should the output head be designed for a regression task with a strictly positive target?
mediumAnswer
- Options: (1) predict and exponentiate — implicit positivity, natural for right-skewed targets; (2) apply or softplus at the output to guarantee positivity; (3) predict a mean of a Gamma/log-normal distribution via a GLM-style head.
- Never use identity output with MSE if y can only be positive — the model will predict negatives during training which then contribute noise.
Check yourself — multiple choice
- Use identity output with MSE
- Predict or apply softplus / exp so predictions are guaranteed positive
- Always add ReLU at the output
- Only use classification instead
Positive target ⇒ trick or softplus/exp at output ⇒ no negative predictions.
#fundamentals#losses
Practise Deep Learning
214 interview questions in this topic.
Related questions
- Why compute softmax + cross-entropy jointly via log-sum-exp?
- How is the output head different for multi-label vs multi-class classification?
- MSE vs MAE for regression — when is each preferred?
- ReLU vs sigmoid vs GELU — when do you use each?
- In one sentence, what is backpropagation?
- State the universal approximation theorem in one sentence and its practical caveat.