How do you handle noisy labels in a supervised dataset?
hardAnswer
- First quantify it: relabel a random sample of a few hundred rows yourself and measure the disagreement rate, which bounds the accuracy any model can reach.
- Then reduce sensitivity: prefer robust losses such as Huber for regression, use label smoothing for classification, and avoid training to zero error since heavy overfitting memorizes the noise.
- Confident learning approaches, such as cleanlab, rank rows whose predicted distribution strongly disagrees with the given label so a human can review the worst offenders.
- Finally, keep a small hand-audited gold set for evaluation — you cannot measure progress against noisy labels.
Check yourself — multiple choice
- Ignore it, models are robust
- Measure the noise rate on a relabelled sample, use robust losses and label smoothing, surface suspect rows, and keep a clean gold evaluation set
- Always drop uncertain rows
- Increase model capacity
Quantify the noise, reduce sensitivity, triage suspect rows, and evaluate on clean labels.
#data-quality#interview-scenarios
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- Your new model has 60% AUC and the baseline had 75%. How do you debug?
- You detect concept drift in production. What do you do?
- How do you train and evaluate a model when labels arrive weeks after predictions?
- Your model scores AUC 0.92 offline but barely helps in production. What are the usual causes?
- How do you decide how often to retrain a supervised model?
- How do you handle class imbalance in a dataset?