How do you evaluate a RAG pipeline end-to-end?
hardAnswer
- Separate the two components: (1) Retrieval eval — recall@k, MRR, nDCG on a labeled query→relevant-docs set.
- (2) Answer eval — faithfulness (is the answer supported by retrieved docs?), answer relevance (does it address the query?), context relevance (are retrieved docs actually relevant?).
- LLM-as-judge frameworks: Ragas, TruLens, DeepEval.
- Human eval on a sample for critical use cases.
- Track both offline metrics and production feedback (thumbs-up rate, complaint rate).
Check yourself — multiple choice
- Only overall accuracy
- Retrieval: recall@k / MRR / nDCG; Answer: faithfulness / relevance via LLM-judge (Ragas / TruLens) + human sample + production signals
- Same as MMLU
- Impossible to evaluate
RAG eval: separate retrieval + answer, LLM-judge + human sample + prod signals.
#rag#evaluation
Practise LLMs & GenAI
214 interview questions in this topic.