Interview: how do you monitor drift in an embedding-based retrieval system?
hardAnswer
- (1) Query distribution drift: KS/PSI on query embedding norms, cluster distribution of queries over time.
- (2) Corpus drift: same on doc embeddings.
- (3) Retrieval quality: NDCG@k on golden query set + regression on click-through rates.
- (4) Semantic drift: cluster queries weekly, compare centroids vs baseline (ARI between weekly clusterings).
- (5) Model drift: recompute embeddings with new version, measure alignment (Procrustes / linear-CKA) to old version.
- Alert on all four.
Check yourself — multiple choice
- Random
- Query embedding distribution drift + corpus drift + retrieval quality (NDCG on golden set + CTR) + weekly clustering ARI + model-version alignment (Procrustes/CKA)
- Only latency
- Not real
Embedding retrieval drift: query/corpus dist + quality + weekly ARI + version alignment.
#interview#monitoring#applications
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Interview: your model's accuracy dropped 15% overnight — how do you diagnose?
- Interview: 'you're asked to segment 10M customers — how?'
- How do you monitor cluster drift over time?
- Interview: 'design an unsupervised fraud detection system'.
- Interview: cluster 5M e-commerce customers for a marketing campaign.
- Interview: production fraud detector — daily volume 10M, current FN too high.