EasyDeepLearn
Supervised Learning · section 16 of 18

Hyperparameter tuning

5 interview questions on hyperparameter tuning, each answered in full. Free to read, no account needed.

Grid search vs random search for hyperparameter tuning — which do you use?

easy
  • Random search almost always beats grid search when you have more than 2-3 hyperparameters.
  • Reason: most hyperparameters have few impactful settings; grid search wastes budget on irrelevant dimensions.
  • Random search covers more diverse combinations for the same number of trials (Bergstra & Bengio, 2012).
  • Use grid only for cheap, low-dim tuning (2-3 params, coarse grid) or as a final refinement around a random-search winner.
  • For real budget, Bayesian optimization / Optuna beats both.
#hyperparameter-tuningPermalink & quiz →

How does Bayesian hyperparameter optimization work at a high level?

hard
  • Maintain a surrogate model of the objective (usually a Gaussian process or Tree-structured Parzen Estimator) as a function of hyperparameters.
  • At each iteration: (1) predict the objective's mean and uncertainty across the search space; (2) use an acquisition function (Expected Improvement, UCB, or TPE-specific rules) to pick the next config — balancing exploration of uncertain regions and exploitation of promising ones; (3) run that config, update the surrogate.
  • Beats random/grid because it *learns* from past trials.
  • Best tools today: Optuna, scikit-optimize, Ax.
#hyperparameter-tuningPermalink & quiz →

What is Optuna's TPE sampler and why is it popular?

hard
  • Tree-structured Parzen Estimator: for each hyperparameter, model p(x  ∣  y  >  y⋅)p(x\; \mid \;y\; > \;y \cdot ) and p(x  ∣  y  ≤  y⋅)p(x\; \mid \;y\; \le \;y \cdot ) — two densities over past trials, split by an objective quantile.
  • Pick the next config to maximize the ratio pgood/pbadp_{\mathrm{good}} / p_{\mathrm{bad}}.
  • Advantages: handles categorical + continuous + conditional hyperparameters natively (unlike GP Bayesian opt), scales to hundreds of trials cheaply, and doesn't require a full covariance matrix.
  • Combined with pruners (asynchronous halving), Optuna is the default open-source hyperparameter tool.
#hyperparameter-tuningPermalink & quiz →

How do Successive Halving / Hyperband / ASHA speed up hyperparameter search?

hard
  • Instead of running every config to completion, allocate a small budget to many configs, kill the worst half (or bottom N/eta), double the budget for survivors, repeat.
  • Successive Halving: fixed budget schedule.
  • Hyperband: run multiple 'brackets' with different tradeoffs between exploration (many configs, small budget) and exploitation (few configs, full budget).
  • ASHA (Asynchronous Successive Halving): parallel version, ideal for distributed clusters.
  • Combined with a smart sampler (random + TPE), Hyperband/ASHA are the fastest generic hyperparameter search algorithms.
#hyperparameter-tuningPermalink & quiz →

How do you tune when you have multiple objectives (accuracy AND latency AND size)?

hard
  • Multi-objective optimization returns a Pareto frontier — configurations not dominated by any other on all objectives.
  • Optuna's NSGA-II or the pymoo library support this natively.
  • In practice you often reduce it to single-objective by: (1) constraining secondary objectives (e.g., 'accuracy s.t. latency < 100ms and model < 100MB') then maximizing the primary; (2) scalarizing with weights; (3) picking a subjective operating point on the returned Pareto frontier.
  • Communicate the tradeoff explicitly to stakeholders — don't ship a single number.
#hyperparameter-tuningPermalink & quiz →

Practise Supervised Learning

All 18 sectionsFlashcardsQuiz