Fine-tuning vs linear probe vs prompt-tuning — when do you use each?
mediumAnswer
- (1) Linear probe: small labeled data + strong SSL encoder → freeze encoder, add linear head.
- Fast, tiny compute.
- (2) Fine-tuning: enough labeled data + task shift → unfreeze last k layers or full network.
- (3) LoRA / adapters: parameter-efficient, when full fine-tune too expensive.
- (4) Prompt-tuning: LLM-specific — learn soft prompts instead of weights.
- Rule: start with linear probe as baseline; escalate to fine-tune only if needed.
Check yourself — multiple choice
- Only full fine-tune
- Linear probe (tiny data, cheap) / fine-tune (lots of labels, task shift) / LoRA-adapters (parameter-efficient) / prompt-tuning (LLM-specific); start with linear probe
- Random
- Not real
Transfer: linear probe / fine-tune / LoRA / prompt-tuning by data + compute.
#representation-learning#applications
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Node2Vec — how does it learn graph node embeddings?
- Self-supervised graph learning — approaches.
- Self-supervised audio — wav2vec 2 and HuBERT.
- How are multimodal embeddings unified across text / image / audio?
- What is a 'foundation model' in the unsupervised sense?
- What is self-supervised learning and why does it matter?