Hyperparameter Tuning Strategies

Systematically searching the hyperparameter space to optimize model generalization without wasting GPU compute.

🟡 intermediate5 min readoptimization

Hyperparameter Tuning optimizes non-trainable configuration settings (learning rate, tree depth, batch size, regularization λ). Grid Search exhaustively evaluates a Cartesian product of predefined values. Random Search samples configurations randomly, outperforming Grid Search when only a few hyperparameters dominate performance (Bergstra & Bengio, 2012). Bayesian Optimization constructs a Gaussian Process surrogate model to balance exploration and exploitation via acquisition functions (Expected Improvement, UCB).

Comparison Matrix

       GRID SEARCH                         RANDOM SEARCH                   BAYESIAN OPTIMIZATION
┌───┬───┬───┐                      ┌───┬───┬───┐                   ┌───┬───┬───┐
│ o │ o │ o │                      │   │ o │   │                   │   │   │ o │ ◄─ Exploitation
├───┼───┼───┤                      ├───┼───┼───┤                   ├───┼───┼───┤
│ o │ o │ o │                      │ o │   │   │                   │ o │   │   │ ◄─ Exploration
├───┼───┼───┤                      ├───┼───┼───┤                   ├───┼───┼───┤
│ o │ o │ o │                      │   │   │ o │                   │   │ o │   │ (Surrogate GP Model)
└───┴───┴───┘                      └───┴───┴───┘                   └───┴───┴───┘
Fixed Grid Points                 Random Distributions             Sequential Informed Search

Strategy	Mechanics	Parallelizable?	Best Used For
Grid Search	Exhaustive Cartesian product of fixed values	Fully Parallel ($O(V^k)$)	Small parameter spaces ($k \le 2$)
Random Search	Uniform/Log-uniform sampling from distributions	Fully Parallel	General baseline tuning ($k \ge 3$)
Bayesian Optimization	Fits Gaussian Process surrogate model + Acquisition Function	Sequential (Batch BO exists)	Expensive training runs (Deep Learning, LLMs)
Hyperband / ASHA	Multi-fidelity sampling + Successive Halving	Fully Parallel	Large neural nets with early stopping

Why Random Search > Grid Search (Bergstra & Bengio)

        GRID SEARCH (3x3 = 9 Trials)               RANDOM SEARCH (9 Trials)
     Max LR  ┌───┬───┬───┐                      Max LR  ┌───┬───┬───┐
       0.1   │ o │ o │ o │                        0.08  │   │ o │   │
       0.01  │ o │ o │ o │                        0.03  │ o │   │   │
       0.001 │ o │ o │ o │                        0.005 │   │   │ o │
             └───┴───┴───┘                              └───┴───┴───┘
              10  50 100                                 12  45  88
             Num Estimators                              Num Estimators
     (Tests ONLY 3 distinct LRs!)               (Tests 9 UNUSED distinct LRs!)

If learning rate controls 90% of model performance and num_estimators controls 10%, Grid Search wastes 6 out of 9 trials re-testing the exact same learning rates. Random Search tests 9 distinct learning rates!

Bayesian Optimization: Surrogate + Acquisition

Surrogate Model: Fits Gaussian Process $\mathcal{GP}(\mu(x), \sigma^2(x))$ over past trials $(x_i, y_i)$.
Acquisition Function: Evaluates next query point $x^* = \arg\max_x a(x)$:
- Expected Improvement (EI): $\mathbb{E}\left[ \max(0, y - y_{\text{best}}) \right]$.
- Upper Confidence Bound (UCB): $\mu(x) + \kappa \cdot \sigma(x)$.

  Surrogate Mean μ(x) + Variance σ(x)          Acquisition Function a(x)
       High Mean  ──► EXPLOITATION               High Variance ──► EXPLORATION

Say this out loud

"Hyperparameter tuning optimizes non-trainable configurations. Random Search outperforms Grid Search in high-dimensional spaces because it tests unique values across critical parameters instead of duplicate grid points. For expensive training runs, we use Bayesian Optimization with Gaussian Processes to select trials balancing exploration and exploitation via Expected Improvement, or Hyperband to prune weak trials early via successive halving."

Follow-ups to expect

Which hyperparameters should be sampled on a logarithmic scale? Learning rate $\alpha$, weight decay $\lambda$, and regularization terms should be sampled log-uniformly (e.g. $10^{-5}$ to $10^{-1}$), because relative scaling changes (0.001 to 0.01) matter far more than linear increments.
What is Optuna? A modern Python hyperparameter optimization framework that uses TPE (Tree-structured Parzen Estimation) for Bayesian sampling combined with automated median pruning for early stopping.

Check yourself

Question 1 of 3

Why does Random Search outperform Grid Search for tuning high-dimensional hyperparameter spaces according to Bergstra & Bengio (2012)?

More in Classical ML

See all →

Bias–Variance Tradeoff4 min Overfitting vs Underfitting3 min Linear Regression4 min