Interview: your image dataset (10M) has near-duplicates — how do you dedup at scale?

hard

Answer

$(1) Perceptual hashing (pHash / dHash / wHash): 64-bit hash \to Hamming distance threshold catches near-duplicates and small edits.$
$Fast, cheap.$
$(2) Deep embedding-based (CLIP / DINOv2) + LSH or FAISS ANN for semantic near-duplicates (same subject, different pose).$
$(3) Combine both — pHash for pixel-level, embeddings for content.$
$Standard in training data prep for image generation (Stable Diffusion 3, Ideogram).$
$Also matters for benchmarking to avoid train-test leak.$

Check yourself — multiple choice

Random
Perceptual hash (pHash) + Hamming for pixel-level near-dup; CLIP/DINOv2 embedding + LSH/FAISS for semantic near-dup — combined; critical for train-test leak in image gen
Just SHA
Not real

Image dedup at scale: pHash + CLIP/DINOv2 + LSH/FAISS combined.

#interview#applications#similarity

214 interview questions in this topic.