In one sentence, what is the Lottery Ticket Hypothesis?
hardAnswer
- Frankle & Carbin (2019): a randomly initialized dense network contains sparse subnetworks ('winning tickets') that — when trained in isolation from their original initialization — reach comparable accuracy in comparable or fewer iterations.
- Motivates pruning research: identify and train these tickets to shrink models dramatically without loss.
Check yourself — multiple choice
- Random init is always optimal
- Dense nets contain sparse subnetworks that can train alone from their init to full accuracy
- Pruning always destroys accuracy
- Only applies to CNNs
Lottery Ticket: sparse subnets with the same init can match dense-net accuracy.
#pruning#training#theory
Practise Deep Learning
214 interview questions in this topic.