How do you spot selection bias in a dataset someone hands you?

hard

Answer

$Ask how a row came to exist, which is a different question from what the row contains.$
$Every dataset is the output of a process that decided what to record, and that process is often correlated with the outcome.$
$Concretely, look for entities that could have been in the data but are not: churned customers absent from a satisfaction table, rejected loan applicants absent from a default model's training set, machines that failed before the sensor was installed.$
$Check whether the sampling rate varies by any variable you care about, and compare the dataset's marginal distributions against a known population where one exists.$
$Then ask what the missingness depends on, because data missing as a function of the unobserved outcome cannot be repaired by imputation and needs a model of the selection itself.$

Check yourself — multiple choice

Check for missing values only
Ask how each row came to be recorded and which eligible entities are absent, whether sampling rates vary by relevant variables, and whether missingness depends on the unobserved outcome
Selection bias shows up as low accuracy
Large datasets are immune

Selection bias lives in the rows that are absent, and outcome-dependent missingness cannot be imputed away.

#sampling#eda

215 interview questions in this topic.