Interview: how would you group log messages from a large distributed system?
hardAnswer
- (1) Parse log: extract template (Drain algorithm, LogPunk) → replace variables (timestamps, IPs, numbers) with placeholders.
- (2) Vectorize templates via TF-IDF or sentence-transformers on remaining text.
- (3) Cluster with HDBSCAN → template categories.
- (4) Anomaly detection: rare template = novel event.
- (5) Interactive drill-down: click cluster → see representative logs + timeline.
- Standard in Splunk, DataDog logs, Elasticsearch's ML.
- Log2Vec / LogBERT are modern DL variants.
Check yourself — multiple choice
- Just grep
- Template extraction (Drain) + vectorize (TF-IDF / sentence-transformers) + HDBSCAN → clusters + anomaly on rare templates + interactive drill-down
- Random
- Not real
Log grouping: Drain template extraction + embed + HDBSCAN + rare-template anomaly.
#interview#applications#clustering
Practise Unsupervised Learning
214 interview questions in this topic.
Related questions
- Interview: 'you're asked to segment 10M customers — how?'
- Interview: cluster 5M e-commerce customers for a marketing campaign.
- Interview: business asks 'why is customer X in Cluster 3?' — how do you explain?
- Interview: how would you cluster 1B rows on a budget?
- Interview: you cluster and 3 clusters look 'right', but 2 mix categories — what next?
- 'Cluster then classify' — when does it help?