EasyDeepLearn
LLMs & GenAI · section 5 of 18

Fine-tuning & adaptation

7 interview questions on fine-tuning & adaptation, each answered in full. Free to read, no account needed.

Fine-tuning vs RAG — which do you use when?

medium
  • Use RAG when the knowledge changes or is large (docs, wikis) — you inject facts at query time without retraining.
  • Use fine-tuning when you need to teach a style, tone, format, or specialized skill (code style, domain jargon).
  • They are complementary: fine-tune for behavior, RAG for knowledge.
  • Prefer RAG first — it is cheaper, updateable, and easier to audit.
#fine-tuning#ragPermalink & quiz →

What is LoRA and why is it popular for fine-tuning LLMs?

hard
  • LoRA (Low-Rank Adaptation) freezes the pretrained weights and injects small trainable low-rank matrices into linear layers.
  • You train only these tiny matrices (often <1% of parameters), so memory and compute drop dramatically while quality stays close to full fine-tuning.
  • Adapters can be swapped at inference for different tasks.
  • QLoRA extends this by fine-tuning on top of a 4-bit quantized base model.
#fine-tuning#lora#peftPermalink & quiz →

What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?

easy
  • SFT takes a pretrained base model and fine-tunes it on high-quality (prompt, completion) pairs written or curated by humans — the format is typically a chat template ('system' + 'user' + 'assistant').
  • Loss: next-token cross-entropy, often masked so only the assistant tokens contribute to the loss.
  • First step of instruction tuning; teaches the model to follow instructions and adopt a conversational format.
  • Typical dataset: 10k-1M curated examples.
  • Followed by RLHF or DPO for preference alignment.
#fine-tuning#alignmentPermalink & quiz →

How are 'reasoning models' like o1 / R1 trained?

hard
  • Combine SFT on long chain-of-thought traces (often self-generated with rejection sampling on math / code datasets) with RL that rewards correct final answers on verifiable tasks (math with numerical checkers, code with unit tests).
  • The model learns to spend variable inference-time compute — long internal 'thinking' before emitting the final answer.
  • DeepSeek-R1 revealed the recipe publicly: GRPO on verifiable rewards + curriculum from easier to harder problems.
#alignment#fine-tuningPermalink & quiz →

Should you fine-tune the LLM for RAG or improve retrieval first?

medium
  • Almost always improve retrieval first — it's cheaper, more auditable, and 80% of RAG problems are retrieval problems (wrong chunks, missing chunks, poor recall).
  • Fine-tune when: (1) the LLM refuses / hallucinates despite good retrieval; (2) the domain has heavy jargon the base LLM stumbles on; (3) you need a specific citation / format style.
  • Debug order: retrieval eval → prompt tuning → reranking → fine-tune generator only if the previous fail.
#rag#fine-tuningPermalink & quiz →

How do you serve many LoRA adapters efficiently?

hard
  • Multi-LoRA serving (S-LoRA, vLLM, Punica): load one base model + hundreds of small LoRA adapters in memory.
  • At request time, apply the requested adapter dynamically during matmul via a specialized kernel that fuses ΔW = B * A into the base W @ x.
  • Adds only KB of weight per adapter.
  • Enables per-tenant / per-task specialization at negligible extra memory.
  • Standard for multi-tenant LLM SaaS with lots of fine-tuned variants.
#serving#peft#lora#inferencePermalink & quiz →

A stakeholder wants the model to 'know our internal docs'. Do you fine-tune or build RAG?

medium
  • RAG, in almost every case.
  • Fine-tuning teaches style, format and task behaviour; it is a poor way to install facts, because the knowledge is baked in at training time and cannot be updated, cited or revoked.
  • RAG keeps documents in a store you can re-index nightly, lets you show sources so the answer is auditable, and handles access control because you can filter retrieval by permission.
  • Fine-tuning becomes the right answer when you need a specific output schema, a domain tone, or lower latency from a smaller model.
  • The two combine well: fine-tune for behaviour, retrieve for facts.
#rag#fine-tuningPermalink & quiz →

Practise LLMs & GenAI

All 18 sectionsFlashcardsQuiz