What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?
easyAnswer
- SFT takes a pretrained base model and fine-tunes it on high-quality (prompt, completion) pairs written or curated by humans — the format is typically a chat template ('system' + 'user' + 'assistant').
- Loss: next-token cross-entropy, often masked so only the assistant tokens contribute to the loss.
- First step of instruction tuning; teaches the model to follow instructions and adopt a conversational format.
- Typical dataset: 10k-1M curated examples.
- Followed by RLHF or DPO for preference alignment.
Check yourself — multiple choice
- SFT skips pretraining
- Fine-tune base LLM on (prompt, completion) pairs with next-token loss on the assistant tokens — first alignment step
- SFT uses RL from scratch
- SFT is only for math
SFT: instruction data → next-token loss on assistant tokens → conversational base.
#fine-tuning#alignment
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How is high-quality instruction-tuning data constructed?
- How are 'reasoning models' like o1 / R1 trained?
- Fine-tuning vs RAG — which do you use when?
- What is LoRA and why is it popular for fine-tuning LLMs?
- What is continual pretraining and when does it beat fine-tuning?
- Describe the full RLHF pipeline in three stages.