How do you monitor a RAG pipeline?

hard

Answer

$(1) Retrieval metrics: recall@k, precision@k, MRR — needs labeled query-doc pairs.$
$(2) Query embedding drift: distribution shift over time.$
$(3) Index freshness: age of most recent doc, ingestion lag.$
$(4) Retrieval quality proxy: does model use retrieved docs (grounding)?$
$(5) End-to-end: answer quality via LLM-judge / golden set.$
$(6) Cost: tokens per query (retrieval + generation).$
$(7) Latency breakdown: embed + search + generate.$
$(8) User: click through, thumbs.$
$Tools: LangSmith, Langfuse, Arize LLM.$

Check yourself — multiple choice

Random
Retrieval (recall/precision/MRR + query drift + index freshness + grounding usage) + E2E (judge/golden) + cost tokens + latency breakdown + user click/thumbs; LangSmith/Langfuse/Arize
Just latency
Not real

RAG monitoring: retrieval + E2E + cost + latency + user; specialized tools.

#llmops#monitoring

215 interview questions in this topic.