How do you detect anomalous predictions in production?
mediumAnswer
- (1) Statistical: z-score of prediction / feature vs recent history.
- (2) Isolation Forest / LOF on request features.
- (3) Autoencoder reconstruction error on inputs.
- (4) Prediction confidence: sudden increase in low-confidence predictions.
- (5) Feature range violations (values outside training distribution).
- (6) Model-specific: attention weights unusual for LLMs. Aggregate to hourly / minute counts + alert on spike.
- Route anomalies to sample store for review.
Check yourself — multiple choice
- Random
- Statistical z-score + Isolation Forest / LOF + autoencoder reconstruction + prediction confidence + feature range + attention anomalies for LLMs; aggregate + alert spike + sample for review
- Not possible
- Just count
Anomaly monitoring: z-score / IF / AE / confidence / range / attention; aggregate + review.
#monitoring
Practise MLOps & Data Quality
215 interview questions in this topic.