How do you train and evaluate a model when labels arrive weeks after predictions?
hardAnswer
- (1) Choose an evaluation delay matching the label horizon — 'test set is data from more than N weeks ago'.
- (2) Never mix current features with future-only labels — check every feature for time-of-availability.
- (3) Use only features available at prediction time (build a snapshot feature store or historical replay).
- (4) Sample the delayed-label distribution: some labels may never arrive; account for censoring (survival analysis) or the label-noise it introduces.
- (5) Monitor a proxy signal (click, conversion micro-events) in the meantime so you notice problems before the true labels land.
Check yourself — multiple choice
- Just wait for all labels before training
- Evaluate with a delay matching the label horizon; use point-in-time features; monitor proxies while waiting
- Use future features
- Delayed labels don't affect anything
Point-in-time features, delayed evaluation, censoring awareness, and proxy signals.
#interview-scenarios#data-quality
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Your new model has 60% AUC and the baseline had 75%. How do you debug?
- You detect concept drift in production. What do you do?
- Your model scores AUC 0.92 offline but barely helps in production. What are the usual causes?
- How do you handle noisy labels in a supervised dataset?
- How do you decide how often to retrain a supervised model?
- How do you handle class imbalance in a dataset?