EasyDeepLearn

All 215 MLOps & Data Quality interview questions

Data drift, class imbalance, monitoring, CI/CD for ML and everything that keeps models alive in production.

Every question below opens on its own page, with a full answer, the formulas written out and a multiple-choice check. Free to read, no account needed.

LessonFlashcardsQuizMock

Data quality & drift

  1. What is data drift and how do you detect it?easy
  2. How is concept drift different from data drift?medium
  3. What does 'data imbalance' mean and why is it a problem?easy
  4. What are common sources of data leakage in an ML pipeline?medium
  5. What should you monitor in a production ML model?easy
  6. How do you validate data quality in an ML pipeline?medium
  7. What is schema drift and how do you detect it?medium
  8. What is label drift and why does it matter?medium
  9. How is Population Stability Index (PSI) computed and interpreted?hard
  10. Why prefer Jensen-Shannon Divergence over KL for drift?hard
  11. What is Maximum Mean Discrepancy (MMD) for drift detection?hard
  12. When is Wasserstein distance appropriate for drift?hard
  13. How do you monitor drift on unstructured data (text / images)?hard
  14. Outliers vs drift — how do you distinguish?medium
  15. How do you handle missing values in a production pipeline?medium
  16. How does imputation cause data leakage?medium
  17. How do you handle label noise in training data?hard
  18. How do you measure and improve annotation quality?medium
  19. What is active learning and when should you use it?medium
  20. What is weak supervision (e.g., Snorkel)?hard
  21. How does time-based leakage happen and how do you prevent it?hard
  22. What is purged cross-validation?hard
  23. SMOTE vs class weights vs threshold tuning — which to use?medium
  24. Why version data (DVC / lakeFS / Delta Lake)?medium
  25. How do you monitor fairness in production?hard
  26. How do you handle PII in ML pipelines?hard
  27. How does selection bias affect ML systems?hard
  28. What is a model feedback loop and why is it dangerous?hard
  29. What is a golden test set and how do you maintain it?medium
  30. What is a data contract?medium
  31. What is point-in-time correctness in feature stores?hard
  32. How do you guarantee online-offline feature consistency?hard
  33. What do you monitor about features (not just models)?medium
  34. How do you structure a progressive rollout with automated guardrails?hard
  35. Why do offline improvements often not translate online?hard
  36. SLO / SLI / SLA for ML services.medium
  37. How do you avoid alert fatigue in ML monitoring?medium
  38. Three pillars of observability applied to ML.medium
  39. How do you monitor feature / prediction freshness?medium
  40. How does shadow evaluation work?medium
  41. What is Canary Analysis (Kayenta / Flagger)?hard
  42. How do you monitor prediction uncertainty in production?hard
  43. How do you monitor model calibration in production?hard
  44. Why monitor per-slice performance?medium
  45. Popular ML observability tools?easy
  46. How do you detect anomalous predictions in production?medium
  47. How do you set thresholds for drift alerts?hard
  48. How do you monitor performance when labels are delayed?hard
  49. How do you monitor drift when data has natural seasonality?hard
  50. What is model decay and how do you measure it?medium
  51. How do you respond to a production ML incident?medium
  52. How do you monitor ML infrastructure cost?medium
  53. How do you handle biased ground truth collection?hard
  54. How do you explain individual predictions in production?medium
  55. How do you compute SHAP at production scale?hard
  56. The four golden signals — Google SRE.easy
  57. RED vs USE vs Golden Signals — which methodology?medium
  58. Why use latency heatmaps instead of averages?medium
  59. Common bad-alert patterns to avoid.medium
  60. How do you monitor GPU utilization?medium
  61. How do you evaluate LLM outputs in production?hard
  62. What are the biases of LLM-as-judge?hard
  63. How do you monitor a RAG pipeline?hard
  64. How do you observe LLM agents?hard
  65. LangSmith / Langfuse — what do they provide?medium
  66. You inherit a model in production with no documentation. What do you check in your first week?medium
  67. Accuracy has probably dropped but labels arrive 60 days late. What can you monitor now?hard
  68. Your drift monitoring fires 40 alerts a day and everyone ignores it. How do you fix it?medium
  69. How do you prove there is no training-serving skew?hard
  70. What is point-in-time correctness, and how does violating it look in practice?hard
  71. Scheduled retraining or triggered retraining?medium
  72. What do you monitor for an LLM feature that you would not monitor for a classifier?hard

Features & pipelines

  1. What problem does a feature store solve?medium
  2. When and how do you retrain a production model?medium
  3. How do you handle high-cardinality categorical features?medium
  4. How do you handle cold-start users / items in production?medium
  5. Online vs offline feature store — architecture.medium
  6. How do you version features?medium
  7. How do you compute real-time features (streaming aggregates)?hard
  8. Orchestration tools for ML pipelines — Airflow vs Prefect vs Dagster.medium
  9. DAG-based vs imperative ML pipelines — tradeoffs.medium
  10. Why must ML pipelines be idempotent?medium
  11. Why partition ML training tables by date?medium
  12. How do you safely backfill features / labels historically?hard
  13. What tests should a training pipeline have?medium
  14. What is an 'asset' in Dagster and why is it useful for ML?hard
  15. Push vs pull materialization for features.medium
  16. ETL vs ELT for ML data — which pattern wins?medium
  17. What is the medallion architecture (bronze/silver/gold)?medium
  18. Common ML pipeline stages in Airflow / Kubeflow.easy
  19. Metaflow — what and when to use it?medium
  20. How does Hydra help ML config management?medium
  21. Continuous training pipeline — what triggers retraining?medium
  22. TFX (TensorFlow Extended) — what is it?medium
  23. ZenML — how is it different from Kubeflow?medium
  24. When is a feature store worth the operational cost?medium

Deployment & experimentation

  1. What is training-serving skew?medium
  2. What is a shadow deployment and how does it differ from A/B testing?medium
  3. What is a canary release for ML models?easy
  4. Batch vs online inference — how do you choose?easy
  5. How do you A/B test a model rigorously?hard
  6. MLflow model registry — what does it provide?medium
  7. What is a model signature and why does it matter?medium
  8. What is a blue-green deployment for ML models?medium
  9. Canary vs blue-green — when to use each?medium
  10. What is champion-challenger pattern?medium
  11. A/B test vs multi-armed bandit — which to use?hard
  12. What is interleaving in recsys A/B testing?hard
  13. What are guardrail metrics in an A/B test?medium
  14. What is a sample ratio mismatch (SRM) alert?hard
  15. What is CUPED and why use it?hard
  16. How does stratified randomization help experiments?hard
  17. Long-term holdout — what and why?hard
  18. How do you meet latency budgets in production ML?hard
  19. How do you control ML serving cost?medium
  20. When is batch inference dramatically cheaper than online?medium
  21. How do you version model artifacts in production?medium
  22. What's a good rollback strategy?medium
  23. How do you handle model errors in production?medium
  24. What is an A/A test and why run it?medium
  25. How do network effects complicate A/B tests?hard
  26. Switchback experiments — when to use?hard
  27. Difference-in-differences for ML experiments — when?hard
  28. How do you compute sample size for A/B?medium
  29. How do you serve multiple models efficiently on shared infrastructure?hard
  30. Why make inference requests idempotent?medium
  31. How to trade model quality vs latency in production?medium
  32. How do you configure autoscaling for ML serving?hard
  33. How do you serve ML models across multiple regions?hard
  34. Shadow deployment, canary, or A/B test — which do you use when?medium
  35. What does a real rollback plan for a model require?medium

Monitoring & observability

  1. Distributed tracing for ML inference — why?medium
  2. Why sample prediction logs and how?medium
  3. What goes into an ML alert runbook?medium
  4. How do you design useful ML dashboards?medium
  5. How long should you retain ML prediction logs?medium
  6. Why is high-cardinality label bad for Prometheus?hard
  7. A model caused a costly wrong decision. What does a useful post-mortem produce?medium

Reproducibility & versioning

  1. What is data / model lineage and why does it matter?medium
  2. What audit trail is required for regulated ML?hard
  3. How do you manage ML config (hyperparameters, thresholds)?medium
  4. MLflow vs Weights & Biases vs Neptune — which to pick?easy
  5. Why containerize ML training + serving?medium
  6. Why isn't setting a seed enough for full reproducibility?hard
  7. How does DVC work?medium
  8. What is lakeFS?medium
  9. What does Delta Lake add over plain Parquet?medium
  10. Weights & Biases vs MLflow — key differences.medium
  11. Git LFS vs DVC — when to use each?medium
  12. How do you make Jupyter notebooks reproducible?medium
  13. Why lock Python dependencies?medium
  14. Docker tag best practices for ML images.medium
  15. Why do some teams use Nix for ML reproducibility?hard
  16. What metadata should be logged per experiment run?medium
  17. What's an artifact store and why separate from model registry?medium
  18. What should you track during LLM fine-tuning?medium
  19. RLHF-specific ops challenges.hard
  20. What do you actually test in a CI pipeline for a model?hard
  21. How do you version prompts and why does it matter as much as model versioning?medium

Infrastructure & serving

  1. Ray for ML — what does it provide?medium
  2. Iceberg vs Delta Lake — which to pick?hard
  3. Kubeflow — what does it provide?medium
  4. NVIDIA Triton Inference Server — what makes it fast?medium
  5. vLLM — why is it fast for LLM serving?hard
  6. PagedAttention — what problem does it solve?hard
  7. Continuous batching vs static batching.hard
  8. TorchServe — when to use it?medium
  9. TensorFlow Serving — when to use it?medium
  10. BentoML — what does it add?medium
  11. KServe (KFServing) — what does it provide?medium
  12. Serverless ML inference — pros / cons?medium
  13. How do you mitigate serverless cold-start latency?medium
  14. ONNX — why use it?medium
  15. TensorRT — what optimizations does it apply?hard
  16. How does quantization affect serving?hard
  17. GPTQ vs AWQ vs SmoothQuant — key differences.hard
  18. FlashAttention — why is it faster?hard
  19. Speculative decoding — how does it accelerate LLMs?hard
  20. Tensor parallelism vs pipeline parallelism vs data parallelism.hard
  21. FSDP / ZeRO — how do they save memory?hard
  22. How do you choose batch size for inference?medium
  23. How do you use spot / preemptible instances safely for ML?medium
  24. How do you deploy ML to edge / mobile?medium
  25. Running ML in the browser — WebAssembly / WebGPU.medium
  26. Which GPU for training vs inference?medium
  27. Network latency in ML serving — how much matters?medium
  28. Kubernetes for ML — key patterns.medium
  29. How can multiple models share a GPU?hard
  30. How do you cache LLM prompts effectively?hard
  31. Vector DB — which one and why?medium
  32. How do you serve many fine-tuned LoRA adapters efficiently?hard
  33. LLM router — what and why?medium
  34. Your model must answer in 50 milliseconds. How do you allocate the budget?medium

LLMOps & advanced

  1. How does DP-SGD work?hard
  2. What is federated learning?hard
  3. How is LLMOps different from traditional MLOps?medium
  4. How do you version prompts in production?medium
  5. Which signals do you wire into an LLMOps pipeline to flag hallucinated responses?hard
  6. Why use hybrid search (dense + sparse) in RAG?medium
  7. How do you chunk documents for RAG?hard
  8. Why re-rank retrieved documents?medium
  9. How do you choose an embedding model?medium
  10. How do you handle embedding model updates without breaking retrieval?hard
  11. How do you defend against prompt injection?hard
  12. How do you monitor for jailbreak attempts?hard
  13. How do you optimize LLM inference cost?medium
  14. What extra content goes in an LLM model card?medium
  15. Common LLM safety evaluation suites.medium
  16. EU AI Act — what does it require?hard
  17. Where is MLOps heading (2025+)?medium

MLOps foundations

  1. What does CI/CD look like for ML?medium
  2. How do you make ML experiments reproducible?medium
  3. What is a model card?easy
  4. Google's MLOps maturity levels (0-2) — what are they?medium
  5. How do you organize ML teams (embedded vs central)?medium

Other topics