What is LoRA and why is it the standard for parameter-efficient fine-tuning?
hardAnswer
- LoRA (Hu et al., 2021) freezes the pretrained weights W and adds a low-rank update ΔW = B * A .
- Only A and B are trained → ~0.1-1% of the total parameters.
- Matches full fine-tuning quality on most tasks at a fraction of memory and disk cost.
- Multiple LoRA adapters can be swapped per task at inference.
- QLoRA quantizes the frozen base to 4-bit and trains LoRA on top — fine-tune 65B models on a single 48GB GPU.
Check yourself — multiple choice
- Same as full fine-tuning
- Freeze W, add low-rank ΔW = B·A trained (~0.1-1% of params); QLoRA quantizes base to 4-bit for cheap giant-model tuning
- Trains all weights
- Only for vision
LoRA: low-rank adapter on frozen backbone → parameter-efficient fine-tuning.
#fine-tuning#transfer-learning#distillation
Practise Deep Learning
214 interview questions in this topic.
Related questions
- What is transfer learning and when is fine-tuning better than feature extraction?
- With a small labelled dataset and a large pretrained backbone, which parameters do you actually train?
- What does temperature do in a softmax?
- What is an EMA of weights and why do modern training recipes use it?
- How does knowledge distillation work?
- What are the main pruning techniques for neural networks?