Interview: how would you cluster 1B rows on a budget?
hardAnswer
- (1) Sample 1M for prototype clustering.
- (2) Feature-engineer + reduce (PCA / hashed features).
- (3) Mini-batch k-means on the sample → get k centroids.
- (4) Extend to 1B via 'assign nearest centroid' in a single map-reduce (Spark, Dask, Ray).
- (5) Or: BIRCH streaming to build micro-clusters, then hierarchical on summaries.
- (6) HDBSCAN via subsample-then-extend heuristic.
- (7) Store cluster IDs alongside data.
- (8) Iterate on 1M until quality is right; only re-run map-reduce on 1B when necessary.
Check yourself — multiple choice
- Random
- Prototype on 1M sample + mini-batch k-means → get centroids + Spark/Dask map-reduce assignment on 1B; BIRCH streaming alt; iterate on sample before full run
- Just k-means
- Not scalable
1B on budget: sample prototype + centroids + Spark assignment; BIRCH streaming alt.
#interview#applications#clustering
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Interview: 'you're asked to segment 10M customers — how?'
- Interview: cluster 5M e-commerce customers for a marketing campaign.
- Interview: how would you group log messages from a large distributed system?
- Interview: business asks 'why is customer X in Cluster 3?' — how do you explain?
- Interview: you cluster and 3 clusters look 'right', but 2 mix categories — what next?
- 'Cluster then classify' — when does it help?