Why partition ML training tables by date?
mediumAnswer
- (1) Prune scans: only read partitions needed for training window (e.g., last 90 days) — 10-100x faster.
- (2) Incremental compute: only recompute affected date's features.
- (3) Time-based split for evaluation is trivial.
- (4) Retention: drop old partitions cheaply.
- (5) Parallel writes per partition.
- Common pattern: folders in S3 / GCS.
- Tools: Delta Lake, Iceberg for transactional table format.
- Consumer reads via SQL WHERE.
Check yourself — multiple choice
- Random
- Prune scans (10-100x faster) + incremental compute + trivial time split + cheap retention + parallel writes; partitions + Delta/Iceberg transactions
- Not helpful
- Just backup
Date partitioning: prune + incremental + split + retention + parallel.
#pipeline#infrastructure
Practise MLOps & Data Quality
215 interview questions in this topic.
Related questions
- Orchestration tools for ML pipelines — Airflow vs Prefect vs Dagster.
- What is an 'asset' in Dagster and why is it useful for ML?
- ETL vs ELT for ML data — which pattern wins?
- What is the medallion architecture (bronze/silver/gold)?
- Metaflow — what and when to use it?
- What are common sources of data leakage in an ML pipeline?