With a small labelled dataset and a large pretrained backbone, which parameters do you actually train?

medium

Answer

$Start with the smallest set that can express the task.$
$Train only a new head first, which is fast, cannot damage the backbone, and gives a baseline within minutes.$
$If that underfits, unfreeze the last block or two, since later layers hold the most task-specific features while early layers encode generic edges and textures that transfer.$
$Use a lower learning rate on unfrozen pretrained weights than on the fresh head, often by a factor of ten, to avoid destroying what pretraining learned.$
$With very little data, a parameter-efficient method such as low-rank adapters gives most of the benefit of full fine-tuning while touching a small fraction of the weights, which also makes the result cheap to store and easy to revert.$

Check yourself — multiple choice

Always fine-tune everything
Head only first, then unfreeze the last blocks with a lower learning rate than the head, and consider low-rank adapters when data is very scarce
Only the first layers
Retrain from scratch

Escalate from a linear head outward, protecting pretrained weights with smaller learning rates.

#transfer-learning#fine-tuning

214 interview questions in this topic.