How does Bayesian hyperparameter optimization work at a high level?

hard

Answer

$Maintain a surrogate model of the objective (usually a Gaussian process or Tree-structured Parzen Estimator) as a function of hyperparameters.$
$At each iteration: (1) predict the objective's mean and uncertainty across the search space; (2) use an acquisition function (Expected Improvement, UCB, or TPE-specific rules) to pick the next config — balancing exploration of uncertain regions and exploitation of promising ones; (3) run that config, update the surrogate.$
$Beats random/grid because it *learns* from past trials.$
$Best tools today: Optuna, scikit-optimize, Ax.$

Check yourself — multiple choice

It's the same as random search
A surrogate model predicts the objective; an acquisition function picks the next config, balancing exploration and exploitation
It requires gradient computations of the metric
It only works for neural nets

Bayesian opt: surrogate model + acquisition function to smartly pick the next trial.

#hyperparameter-tuning

215 interview questions in this topic.