Why should preprocessing live inside a scikit-learn Pipeline rather than being applied manually before?
easyAnswer
- Pipelines guarantee that every preprocessing step is fit on the *training fold* only (during CV, hyperparameter search, and grid search) and applied to validation/test.
- Manual preprocessing before splitting is the classic source of leakage: a StandardScaler fit on the full data has already seen the test set's variance.
- Pipelines also encapsulate the whole model into one deployable object — pickle it, deploy it, and inputs get transformed identically at serving time.
- Non-negotiable for reproducibility and safety.
Check yourself — multiple choice
- Pipelines are slower
- They fit preprocessing only on training folds during CV — the standard way to prevent leakage
- Pipelines don't work with cross-validation
- Preprocessing must always be manual
Pipelines fit preprocessing per fold ⇒ no leakage + one deployable artifact.
#pipelines#data-quality
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- You did StandardScaler().fit_transform(X) then split into train/test. Why is this wrong?
- How do you handle class imbalance in a dataset?
- What is target leakage and how do you prevent it?
- You use stratified k-fold on a dataset with duplicate customer records — why is it wrong?
- How do you prevent leakage in target encoding?
- MCAR, MAR, MNAR — what are these and why do they matter?