EasyDeepLearn

Write out the softmax function and its main property.

easy

Answer

  • softmax⁡(z)k  =  exp⁡(zk)  /  sumj\operatorname{softmax}(z)k\; = \;\operatorname{exp}(z_{k})\; / \;\mathrm{sum}_{j} exp⁡(zj)\operatorname{exp}(z_{j}).
  • It maps a vector of real logits to a probability distribution: values in (0, 1) that sum to 1.
  • It's invariant to adding a constant to all logits (that's why we subtract max(z) for numerical stability).
  • The winner is the argmax; adjacent logits produce similar probabilities so the mapping is smooth and differentiable — perfect for gradient descent.
Check yourself — multiple choice
  • Softmax sums to zero
  • softmax⁡(z)k  =  exp⁡(zk)  /  sumj\operatorname{softmax}(z)k\; = \;\operatorname{exp}(z_{k})\; / \;\mathrm{sum}_{j} exp⁡(zj)\operatorname{exp}(z_{j}); it maps logits to a probability distribution
  • Softmax always outputs one-hot vectors
  • Softmax is not differentiable

softmax = normalized exponentials; smooth, differentiable, sums to 1.

#logistic-regression#classification

Practise Supervised Learning

215 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions