How do Successive Halving / Hyperband / ASHA speed up hyperparameter search?
hardAnswer
- Instead of running every config to completion, allocate a small budget to many configs, kill the worst half (or bottom N/eta), double the budget for survivors, repeat.
- Successive Halving: fixed budget schedule.
- Hyperband: run multiple 'brackets' with different tradeoffs between exploration (many configs, small budget) and exploitation (few configs, full budget).
- ASHA (Asynchronous Successive Halving): parallel version, ideal for distributed clusters.
- Combined with a smart sampler (random + TPE), Hyperband/ASHA are the fastest generic hyperparameter search algorithms.
Check yourself — multiple choice
- Run every configuration to full training
- Multi-fidelity: give few resources to many configs, kill the worst, promote survivors, repeat
- Same as random search
- Only used for regression
SH/Hyperband/ASHA: multi-fidelity — allocate budget iteratively to the best-performing configs.
#hyperparameter-tuning
Practise Supervised Learning
215 interview questions in this topic.
Related questions
- How do you choose the regularization strength (lambda / alpha)?
- How do you choose k in k-Nearest Neighbours?
- What are the most impactful hyperparameters of a decision tree?
- Which Random Forest hyperparameters actually matter for tuning?
- What is the role of the learning rate (shrinkage) in GBM?
- Which XGBoost hyperparameters have the biggest impact, and in what tuning order?