What is LoRA and why is it popular for fine-tuning LLMs?
hardAnswer
- LoRA (Low-Rank Adaptation) freezes the pretrained weights and injects small trainable low-rank matrices into linear layers.
- You train only these tiny matrices (often <1% of parameters), so memory and compute drop dramatically while quality stays close to full fine-tuning.
- Adapters can be swapped at inference for different tasks.
- QLoRA extends this by fine-tuning on top of a 4-bit quantized base model.
Check yourself — multiple choice
- LoRA updates all model parameters
- LoRA trains small low-rank matrices while freezing the base model
- LoRA increases memory usage over full fine-tuning
- QLoRA requires a full-precision base model
LoRA = low-rank adapters injected in linear layers; base weights are frozen.
#fine-tuning#lora#peft
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- How do you serve many LoRA adapters efficiently?
- Fine-tuning vs RAG — which do you use when?
- What is continual pretraining and when does it beat fine-tuning?
- What is supervised fine-tuning (SFT) and where does it fit in the alignment pipeline?
- How is high-quality instruction-tuning data constructed?
- How are 'reasoning models' like o1 / R1 trained?