EasyDeepLearn

How do you compute the parameter count of a linear layer and a convolution?

medium

Answer

  • Linear(in, out): in * out + out (weights + bias).
  • Conv2d(Cin,  Cout,  kernel=k,  stride=s,  pad=p)\mathrm{Conv2d}(C_{\mathrm{in}}, \;C_{\mathrm{out}}, \;\mathrm{kernel} = k, \;\mathrm{stride} = s, \;\mathrm{pad} = p): Cin  ⋅  Cout  ⋅  k  ⋅  k  +  CoutC_{\mathrm{in}}\; \cdot \;C_{\mathrm{out}}\; \cdot \;k\; \cdot \;k\; + \;C_{\mathrm{out}}.
  • Multi-head attention with hidden d and n heads: 4  ⋅  d24\; \cdot \;d^{2} (Q, K, V, out projections) + biases.
  • MLP block: 2  ⋅  d  ⋅  dffn2\; \cdot \;d\; \cdot \;d_{\mathrm{ffn}} (up + down projection).
  • Practical parameter count for a transformer layer ≈ 12  ⋅  d212\; \cdot \;d^{2} (attn  +  MLP  with  dffn  =  4d)(\mathrm{attn}\; + \;\mathrm{MLP}\;\mathrm{with}\;d_{\mathrm{ffn}}\; = \;4d).
Check yourself — multiple choice
  • Only the number of layers matters
  • Linear: in*out + out; Conv: Cin⋅Cout⋅k2  +  Cout\mathrm{Cin} \cdot \mathrm{Cout} \cdot k^{2}\; + \;\mathrm{Cout}; transformer layer ≈ 12·d2d^{2}
  • Bias never counts
  • Attention has no parameters

Basic counts + ~12·d2d^{2} per transformer layer (attention  +  MLP  with  dffn=4d)(\mathrm{attention}\; + \;\mathrm{MLP}\;\mathrm{with}\;d_{\mathrm{ffn}} = 4d).

#fundamentals#architectures

Practise Deep Learning

214 interview questions in this topic.

Read the lessonFlashcardsTake the quiz

Related questions