State the universal approximation theorem in one sentence and its practical caveat.
mediumAnswer
- A feedforward network with a single hidden layer of finite width can approximate any continuous function on a compact domain to arbitrary accuracy — but the theorem says nothing about width required, training feasibility, or generalization.
- In practice depth is much more parameter-efficient than width, which is why deep networks dominate.
Check yourself — multiple choice
- A single wide hidden layer can approximate any continuous function on a compact set (existence only, not efficiency)
- Every neural net trained by SGD reaches the global optimum
- Depth is always useless if you have enough width
- The theorem gives an explicit width bound
Existence result: says approximation is possible, not efficient or trainable.
#fundamentals#theory
Practise Deep Learning
214 interview questions in this topic.
Related questions
- ReLU vs sigmoid vs GELU — when do you use each?
- In one sentence, what is backpropagation?
- Why does stacking linear layers without nonlinearity collapse to a single linear layer?
- Depth vs width — which do you scale first?
- Why can't you initialize a neural net with all zeros?
- What is a computation graph and why does autograd care?