Which signals do you wire into an LLMOps pipeline to flag hallucinated responses?

hard

Answer

$(1) Self-consistency: sample multiple responses; disagreement = uncertainty.$
$(2) Confidence via log probs (low prob tokens = likely halluc).$
$(3) Retrieval-grounding check: does output contain claims supported by retrieved context?$
$(4) Factuality LLM (e.g., FactScore, HHEM) as judge.$
$(5) NER / claim extraction + verifier lookup.$
$(6) User feedback loops (thumbs down).$
$(7) Post-hoc: check output against knowledge base.$
$Route flagged responses to human.$
$LLM-judge for scale.$

Check yourself — multiple choice

Random
Self-consistency (sample+disagree) + log prob confidence + retrieval-grounding check + factuality LLM (FactScore/HHEM) + claim extraction+verifier + user thumbs + post-hoc KB check; flag + human
Impossible
Not real

Halluc: consistency + log prob + grounding + factuality + claim + user + KB.

#llmops#safety

215 interview questions in this topic.