What do you monitor for an LLM feature that you would not monitor for a classifier?
hardAnswer
- Output-side quality signals, because there is no single correct label to compare against.
- Track refusal and empty-response rates, which spike when a prompt template or provider changes.
- Track groundedness for retrieval features, meaning the fraction of claims supported by the retrieved context, usually with an automated judge validated against human labels.
- Track token counts and cost per request, since these drift with user behaviour and directly hit the budget.
- Track latency to first token separately from total latency because they have different causes.
- Log the prompt version, model version and retrieved document identifiers with every request, since without them a regression is undebuggable.
- And sample real conversations for human review on a fixed cadence, which remains the highest-signal monitoring available.
Check yourself — multiple choice
- Nothing different, accuracy still applies
- Refusal rates, groundedness against retrieved context, token cost, time to first token, versioned prompt and retrieval logs, plus a sampled human review cadence
- Only GPU utilization
- LLM features cannot be monitored
With no single ground truth, monitoring shifts to grounding, refusals, cost and sampled human review.
#llmops#monitoring#observability
Practise MLOps & Data Quality
215 interview questions in this topic.