DAG-based vs imperative ML pipelines — tradeoffs.
mediumAnswer
- DAG-based (Airflow, Kubeflow): explicit dependency graph; scheduler manages execution + parallelism; better observability + retries.
- Downside: harder to develop locally + parametrize.
- Imperative (Python scripts): easy to write + debug; poor scalability + retries.
- Modern middle ground: Prefect / Dagster / Metaflow decorate Python functions as tasks, produce DAG at runtime.
- Ray for parallel Python.
- Choose based on team maturity + scale.
Check yourself — multiple choice
- Random
- DAG (Airflow/Kubeflow): explicit graph + scheduler + observable; script: easy dev + poor retry; middle ground (Prefect/Dagster/Metaflow) decorate Python → runtime DAG; Ray for parallel
- Same
- Not real
DAG vs imperative: modern middle ground decorates Python → runtime DAG.
#pipeline
Practise MLOps & Data Quality
215 interview questions in this topic.