How do you evaluate LLM outputs in production?

hard

Answer

$(1) LLM-as-judge: strong model rates outputs on rubric (accuracy, helpfulness, safety) — cheap + fast, but bias toward similar-style output.$
$(2) Golden set: hand-labeled Q&A / rubric — expensive but reliable.$
$(3) Reference-based metrics (BLEU / ROUGE / BERTScore) for translation / summarization.$
$(4) Automatic: exact match / regex / JSON validity.$
$(5) Human eval periodic sample.$
$(6) Downstream signals: click / thumbs up / task completion.$
$(7) Guardrails: refusal, PII, harmful content detection.$
$Combine multiple.$

Check yourself — multiple choice

Random
LLM-judge (rubric, cheap, style bias) + golden set (labor) + reference (BLEU/ROUGE/BERTScore) + automatic (exact/regex/JSON) + human sample + downstream (click/thumbs) + guardrails; combine
Just accuracy
Not real

LLM eval: judge + golden + reference + auto + human + downstream + guardrails.

#llmops#monitoring

215 interview questions in this topic.