Your RAG system gives a wrong answer. How do you find out which stage failed?

hard

Answer

$Separate retrieval from generation, because the fixes are completely different.$
$Log the retrieved chunks for the query and read them.$
$If the correct passage is absent, it is a retrieval failure: look at chunking, the embedding model, hybrid search, or the top-k cutoff.$
$If the passage is present but the model ignored or misread it, it is a generation failure: look at prompt ordering, context length, or the instruction to ground answers in the sources.$
$Track the two as separate metrics — retrieval recall at k, and faithfulness given correct context — otherwise you will keep tuning the wrong half.$

Check yourself — multiple choice

Just change the model
Inspect the retrieved chunks: absent passage means a retrieval failure, present but ignored means a generation failure — measure both separately
It is always the prompt
RAG cannot be debugged

Retrieval recall and grounded faithfulness must be measured separately to localize the fault.

#rag#evaluation

214 interview questions in this topic.