What should you track during LLM fine-tuning?
mediumAnswer
- (1) Loss per step (train + val).
- (2) Perplexity on held-out.
- (3) Task-specific metrics: BLEU / ROUGE / accuracy / rubric score.
- (4) Sample outputs at regular intervals — read them.
- (5) LR schedule + gradient norm + weight norm.
- (6) GPU utilization + throughput (tokens/sec).
- (7) OOM / crash logs.
- (8) Data mixture per step (for multi-source).
- (9) Reference model divergence (KL) if RLHF/DPO.
- (10) Downstream benchmark (MMLU / HellaSwag / your-eval) at checkpoints.
- Track via W&B / MLflow.
Check yourself — multiple choice
- Just loss
- Loss/perplexity + task metrics + sample outputs + LR/grad/weight norm + GPU/throughput + OOM logs + data mixture + KL ref divergence + downstream bench at ckpts; W&B/MLflow
- Random
- Not real
Fine-tune tracking: loss + task + samples + norms + GPU + KL + downstream.
#llmops#reproducibility
Practise MLOps & Data Quality
215 interview questions in this topic.