How do you monitor drift on unstructured data (text / images)?

hard

Answer

$(1) Compute embeddings from pretrained model (CLIP for images, sentence-BERT for text).$
$(2) Reduce to low-D via PCA / UMAP for viz.$
$(3) Compute MMD / energy distance / Wasserstein between train and current embeddings.$
$(4) Alert on threshold breach.$
$(5) Sample outliers with high per-example score for human review.$
$Complementary: monitor prediction confidence distribution + per-cluster prevalence.$

Check yourself — multiple choice

Cannot monitor
Embed with CLIP / sentence-BERT → MMD / energy / Wasserstein on embeddings + PCA/UMAP viz + outlier review + monitor prediction confidence
Random
Just count rows

Unstructured drift: embed + distance metric + outlier review.

#monitoring#data-quality

215 interview questions in this topic.