EasyDeepLearn
MLOps & Data Quality · section 4 of 8

Monitoring & observability

7 interview questions on monitoring & observability, each answered in full. Free to read, no account needed.

Distributed tracing for ML inference — why?

medium
  • Single request spans: feature store fetch → preprocess → model call → postprocess → API response.
  • Tracing (Jaeger, OpenTelemetry) shows per-span latency + errors + attributes.
  • Debugging: p99 latency dominated by feature fetch (not model) reveals where to optimize.
  • Correlated failure: one downstream service down → trace shows which span failed.
  • Modern: LLM traces per prompt token / retrieval hop (LangSmith, Langfuse, Weights & Biases Traces).
#observability#infrastructurePermalink & quiz →

Why sample prediction logs and how?

medium
  • High-volume services (millions QPS) can't log every prediction.
  • Sample: (1) uniform 1% for baseline monitoring.
  • (2) stratified by class or region for balanced view.
  • (3) 100% sample of errors + top-K uncertain predictions (importance sampling for anomalies).
  • (4) tail sample: p99 latency requests.
  • Storage: sample to cold store (Parquet on S3); realtime metrics from Prometheus counters (not sampled).
  • Trade-off: log volume cost vs debug capability.
#observabilityPermalink & quiz →

What goes into an ML alert runbook?

medium
  • (1) Alert description + severity.
  • (2) Impact assessment: what's affected?
  • (3) Diagnostic queries: SQL / Grafana links.
  • (4) Common causes + likely fixes.
  • (5) Escalation path if diagnostics fail.
  • (6) Rollback / mitigation steps.
  • (7) Post-mortem template link.
  • Stored in wiki / Runhouse / PagerDuty; linked from every alert.
  • Rule: every alert must have a runbook; if you can't write one, the alert isn't ready to fire.
#observabilityPermalink & quiz →

How do you design useful ML dashboards?

medium
  • (1) Top-of-funnel: at-a-glance health (traffic + errors + latency + drift).
  • (2) Drill-down per model / feature / slice.
  • (3) One 'page' per concern (drift, accuracy, infra, cost).
  • (4) Comparison: current vs baseline / last week.
  • (5) Annotations for deploys / incidents.
  • (6) Alerts embedded (fires on this graph).
  • (7) Links to runbooks.
  • (8) Own audience: engineer dashboards ≠ exec dashboards.
  • Bad dashboard: 40 charts nobody reads; good: 5 charts driving action.
#observabilityPermalink & quiz →

How long should you retain ML prediction logs?

medium
  • Depends on: (1) regulatory (banking / medical: 7-10 years).
  • (2) debug window (recent logs hot 30 days; older cold storage).
  • (3) retraining data (labeled predictions kept indefinitely).
  • (4) storage cost balance.
  • Typical: (a) hot: 30 days (Elasticsearch / Loki).
  • (b) warm: 6-12 months (Parquet on S3).
  • (c) cold: archive to Glacier for 5-10 years compressed.
  • Aggregate metrics (Prometheus) forever at reduced resolution.
#observabilityPermalink & quiz →

Why is high-cardinality label bad for Prometheus?

hard
  • Each unique label combination = separate time series.
  • Explosive: latency{user_id=..., request_id=...} → millions of series → memory blowup.
  • Rule: labels ≤ 10 unique values, aggregate at query time.
  • High-cardinality data belongs in logs / traces, not metrics.
  • Modern alternatives: VictoriaMetrics / Mimir handle better; ClickHouse for OLAP-style.
  • Or use histograms + percentiles instead of raw values.
#observability#infrastructurePermalink & quiz →

A model caused a costly wrong decision. What does a useful post-mortem produce?

medium
  • A timeline anchored in evidence, which is only possible if predictions, inputs and versions were logged, and a first finding of a missing log is itself the most valuable output.
  • Then the actual causal chain, which for model incidents is usually upstream: a schema change, a feature pipeline default, or drift nobody was alerted on, rather than the model being wrong in an interesting way.
  • Identify the detection gap, meaning how long the problem ran before anyone noticed, since reducing time to detection is generally worth more than preventing that specific cause.
  • Produce concrete actions with owners, weighted towards guardrails and validation rather than promises of vigilance.
  • And keep it blameless, because engineers who fear the process stop reporting the near misses that are your cheapest information.
#mlops#observabilityPermalink & quiz →

Practise MLOps & Data Quality

All 8 sectionsFlashcardsQuiz