Hyperparameter tuning and AutoML

In short

Hyperparameters are the settings chosen *before* training (learning rate, tree depth, number of layers, regularisation strength); parameters are the weights learned *during* training. Hyperparameter tuning is the search for the setting that minimises validation error, and AutoML automates the whole pipeline - preprocessing, feature engineering, model choice, tuning and sometimes architecture design. It decides whether a model that is theoretically right actually performs, so it carries marks in machine-learning and data-science papers (AICTE model curriculum AI&DS, UGC CBCS electives) and appears in GATE DA (Data Science and AI, introduced 2024) through cross-validation, bias-variance and model selection. In Indian hiring - TCS, Infosys, Wipro, Mu Sigma, Fractal, Swiggy, Zomato, PhonePe, Razorpay, Flipkart, Jio, product teams at Google and Microsoft India - the standard interview question is "how did you tune it, and how do you know you did not overfit the validation set?"

Explained

Parameters versus hyperparameters, and the three-way data split

• Parameter: learned by the optimiser from data - weights and biases of a network, split thresholds in a tree, support-vector coefficients.
• Hyperparameter: fixed by the designer before or between training runs - cannot be obtained by gradient descent on the training loss (doing so would always pick zero regularisation).
• Model hyperparameters control capacity (depth, number of hidden units, kernel, max_depth, n_estimators); algorithm/optimiser hyperparameters control the training process (learning rate, batch size, momentum, epochs, weight decay, dropout rate).
• Training set fits parameters; validation set selects hyperparameters; test set is touched once, for the final honest estimate.
• Repeatedly tuning on the test set makes it a second validation set - the reported accuracy becomes optimistic. This is selection bias / validation overfitting.
• Nested cross-validation: inner loop tunes, outer loop estimates generalisation - the correct answer when the dataset is too small for a held-out test set.

Validation strategies that tuning rests on

• Hold-out: single split (e.g. 70/15/15). Cheap, high variance on small data.
• k-fold CV: data split into k folds; each fold is validation once. k = 5 or 10 is standard; error = mean of k scores, with the standard deviation reported.
• Stratified k-fold: preserves class proportions - mandatory for imbalanced problems such as fraud or churn.
• Leave-one-out (LOOCV): k = n; almost unbiased but very high variance and n fits.
• Time-series / rolling-origin CV: train on the past, validate on the future; never shuffle. Required for demand, load or price forecasting.
• Group k-fold: keeps all rows of one patient, customer or device in one fold to stop identity leakage.
• Leakage rules: fit scalers, imputers, encoders, SMOTE and feature selection *inside* each fold - use Pipeline + GridSearchCV, never scale the full dataset first.

Search strategies

1. Manual / graduate-student search: tune by intuition, one knob at a time. Not reproducible; still the most common in practice.
2. Grid search: full Cartesian product of discrete values. Exhaustive, embarrassingly parallel, but cost is exponential in dimensions - the curse of dimensionality. With 5 values on 6 hyperparameters and 5-fold CV that is 5⁶ × 5 = 78,125 fits.
3. Random search: sample each hyperparameter from a distribution (log-uniform for learning rate and regularisation). Bergstra and Bengio (2012) showed it beats grid search at equal budget because only a few hyperparameters matter - the low effective dimensionality argument; a grid wastes trials repeating the same value of the important knob.
4. Bayesian optimisation (SMBO): build a cheap surrogate of the validation-error landscape (Gaussian process, or TPE - Tree-structured Parzen Estimator in Optuna/Hyperopt, or random forest in SMAC), then pick the next point by maximising an acquisition function - Expected Improvement, Probability of Improvement, or Upper/Lower Confidence Bound - which balances exploitation (near the best point) against exploration (where variance is high). Sequential, so less parallel, but far fewer trials.
5. Successive halving (SHA): start n configurations with a small budget (few epochs, subsample, few trees), keep the best 1/η, multiply the budget by η, repeat. Hyperband runs several brackets of successive halving with different starting n to hedge the budget-vs-breadth trade-off; ASHA/BOHB add asynchrony and a Bayesian model.
6. Population-based: genetic algorithms and evolutionary search (TPOT evolves whole pipelines), Population Based Training (PBT) copies weights from good workers and perturbs their hyperparameters mid-training, and CMA-ES for continuous spaces.
7. Gradient-based / hypergradient: differentiate validation loss with respect to continuous hyperparameters; used in DARTS-style architecture search.
8. Early stopping as a tuner: monitor validation loss with patience, restore the best weights - this tunes the "number of epochs" hyperparameter for free.

What to tune, and typical ranges

• Order of importance for deep nets: learning rate ≫ batch size and schedule > regularisation > architecture width/depth > initialisation.
• LR schedules: step decay, exponential, cosine annealing, one-cycle, ReduceLROnPlateau, warm-up for transformers.
• Bias-variance link: underfitting (high train and validation error) means raise capacity or lower regularisation; overfitting (low train, high validation error) means more data, stronger regularisation, dropout, early stopping or less capacity.

AutoML: scope, components, tools

• Definition: systems that automate the end-to-end supervised pipeline - data cleaning, encoding, imputation, feature selection/engineering, model selection, hyperparameter tuning, ensembling and deployment-ready export - so that a non-expert can produce a competitive model.
• CASH problem (Combined Algorithm Selection and Hyperparameter optimisation): search jointly over the discrete choice of algorithm and its own conditional hyperparameter space; handled with a hierarchical/conditional search space.
• Meta-learning / warm start: Auto-sklearn stores meta-features (rows, columns, class ratio, skew) of past datasets and seeds the search with configurations that worked on similar ones - "learning to learn".
• Ensembling: most AutoML gains come from stacking or greedy ensemble selection over the trial models, not from one winner.
• Neural Architecture Search (NAS): search space (cells, operations) + strategy (RL controller, evolution, DARTS gradient relaxation) + performance estimation (weight sharing, one-shot supernet, proxy tasks). Classic NAS was enormously expensive; weight-sharing and DARTS cut it to GPU-days. Hardware-aware NAS adds latency and memory objectives (MnasNet, EfficientNet scaling).
• Libraries: Optuna, Hyperopt, Ray Tune, scikit-optimize, Keras Tuner (tuners), Auto-sklearn, TPOT, AutoGluon, H2O AutoML, FLAML, AutoKeras, PyCaret (AutoML); cloud - Vertex AI AutoML, Amazon SageMaker Autopilot/AMT, Azure AutoML, DataRobot.
• Tracking: MLflow, Weights & Biases, TensorBoard HParams - log every trial's config, seed, metric and artefact; an untracked search is not reproducible.
• Limits: compute and rupee cost, carbon footprint, opaque pipelines, poor handling of domain-specific feature engineering and of distribution shift, risk of silent leakage if the user feeds a badly constructed table, and weak support for causal or fairness constraints. AutoML automates search, not problem formulation.

Practice, cost and Indian context

• Budgeting: fix a trial budget and a per-trial timeout first; use a cheap proxy (subsample, fewer epochs, smaller image size) for the broad sweep, then refine the top few on full data.
• Multi-objective / multi-fidelity: optimise accuracy *and* latency, size or inference cost - Optuna and Ray Tune support Pareto fronts; essential for on-device models in Indian fintech and edge apps.
• Reproducibility: fix seeds for data split, initialisation and sampler; report the sampler and the budget. Scores differing in the third decimal across seeds are noise, not improvement.
• Compliance: the Digital Personal Data Protection Act, 2023 governs personal data used in training and tuning - purpose limitation and minimisation apply to experiment copies of data too (its rules were notified in 2025; check the current notified text and timelines before relying on dates). RBI's guidance for banks and NBFCs expects model governance, documented validation and auditability, so tuning logs are evidence; SEBI requires disclosure of AI/ML use by intermediaries. India's AI governance framework is advisory rather than a statute as of late 2026 - say so rather than quoting an "AI Act".
• Cost in rupees: a cloud GPU (A100-class) rents for roughly ₹200-400 per hour in India depending on provider and commitment - rates change, so quote the current price list; a 200-trial sweep of 20 minutes each is about 67 GPU-hours, so the search must be budgeted like any other resource.

Key points

• Hyperparameter - set before training, not learned by the optimiser; parameter - learned weights.
• Validation set - selects hyperparameters; test set - used once, otherwise the estimate is optimistic.
• k-fold CV - average of k held-out scores; stratify for imbalance, group for repeated entities, rolling for time series.
• Grid search - exhaustive, parallel, cost grows exponentially with the number of hyperparameters.
• Random search - equal or better at the same budget, because only a few hyperparameters truly matter.
• Bayesian optimisation - surrogate model plus acquisition function (EI, UCB); fewest trials, but sequential.
• TPE - Optuna's default sampler; models p(config | good) vs p(config | bad) instead of a Gaussian process.
• Successive halving / Hyperband - allocate little budget to many configs, kill the weak early, promote the rest.
• Early stopping with patience - tunes epoch count and prevents overfitting at no extra cost.
• Log-uniform sampling - correct for learning rate, C, λ, eta; uniform sampling wastes trials there.
• CASH problem - algorithm choice and its hyperparameters searched jointly in a conditional space.
• AutoML stack - preprocessing + model selection + tuning + ensembling, often meta-learning-warm-started.
• NAS - search space + search strategy + performance estimation; DARTS and weight sharing made it affordable.
• Pipeline-inside-CV - scaling, imputation and SMOTE must be fitted per fold or results leak.
• Multi-fidelity - cheap proxy first (subsample, few epochs), full fidelity only for finalists.
• Nested CV - inner loop tunes, outer loop reports; the honest protocol on small data.
• Reproducibility - seed, sampler, budget and trial log; third-decimal gains are usually noise.
• AutoML limits - cost, opacity, no domain feature engineering, no problem formulation, DPDP Act duties still apply.

Formulas and examples

Example 1 - grid versus random cost. An XGBoost grid: eta {0.01, 0.05, 0.1}, max_depth {3, 5, 7, 9}, subsample {0.6, 0.8, 1.0}, colsample {0.6, 0.8, 1.0}, with 5-fold CV.

• Combinations = 3 × 4 × 3 × 3 = 108; fits = 108 × 5 = 540.
• At 12 s per fit on one core, serial time = 6,480 s = 1.8 hours; on 8 parallel workers ≈ 13.5 minutes.
• A 60-trial random search costs 60 × 5 = 300 fits = 1 hour serial, and explores continuous eta rather than three fixed values.

Example 2 - how many random trials. Probability of landing in the top 5% of the configuration space.

• p = 0.05, n = 60 → P = 1 − 0.95⁶⁰. ln(0.95) = −0.051293; × 60 = −3.0776; e^−3.0776 = 0.0461 → P = 95.4%.
• For P = 0.99: n ≥ ln(0.01)/ln(0.95) = (−4.6052)/(−0.051293) = 89.8 → 90 trials.
• For the top 1% (p = 0.01) at P = 0.95: n ≥ ln(0.05)/ln(0.99) = (−2.9957)/(−0.0100503) = 298.1 → 299 trials. Cost grows as 1/p, not with the number of dimensions - the key advantage over grid search.

Example 3 - reading a 5-fold CV result. Two configurations on a churn dataset, F1 per fold:

• Mean of A: (0.812+0.798+0.825+0.805+0.810) = 4.050 → 0.8100. Mean of B: (0.818+0.790+0.830+0.796+0.816) = 4.050 → 0.8100.
• Identical means; A's spread is smaller, so A is the safer pick. Difference of 0.000 ± 0.009 - any "winner" chosen on the third decimal here would be noise. Prefer the simpler/faster model when scores tie.

Example 4 - successive halving budget. n = 81 configurations, η = 3, starting budget 1 epoch.

• Rounds: 81 configs × 1 epoch = 81; 27 × 3 = 81; 9 × 9 = 81; 3 × 27 = 81; 1 × 81 = 81.
• Rounds = log₃(81) + 1 = 5; total = 405 epoch-units.
• Training all 81 for the full 81 epochs would cost 81 × 81 = 6,561 epoch-units - successive halving is about 16× cheaper for the same final winner budget, assuming early performance ranks configurations usefully.

Example 5 - rupee cost of a sweep. 150 Optuna trials, average 8 minutes per trial, 4 parallel GPU workers at ₹250 per GPU-hour.

• Total GPU time = 150 × 8 = 1,200 GPU-minutes = 20 GPU-hours; cost = 20 × 250 = ₹5,000 (wall-clock ≈ 20/4 = 5 hours).
• With Hyperband pruning cutting average trial time to 3 minutes: 150 × 3 = 450 min = 7.5 GPU-hours = ₹1,875 - a 62.5% saving. Rates vary by provider; use the current price list.

import optuna
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

def objective(trial):
    params = dict(
        learning_rate=trial.suggest_float("learning_rate", 1e-3, 3e-1, log=True),
        max_leaf_nodes=trial.suggest_int("max_leaf_nodes", 15, 255, log=True),
        l2_regularization=trial.suggest_float("l2", 1e-8, 10.0, log=True),
        max_iter=600, early_stopping=True, random_state=42,
    )
    pipe = Pipeline([                      # fitted inside each fold - no leakage
        ("imp", SimpleImputer(strategy="median")),
        ("sc", StandardScaler()),
        ("clf", HistGradientBoostingClassifier(**params)),
    ])
    return cross_val_score(pipe, X_tr, y_tr, cv=cv, scoring="f1").mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=42),
                            pruner=optuna.pruners.HyperbandPruner())
study.optimize(objective, n_trials=150, timeout=3600)
print(study.best_params, round(study.best_value, 4))   # test set touched only now

Where it is used

• Fintech and credit risk (PhonePe, Razorpay, Paytm, HDFC/ICICI analytics, NBFC scorecards): tuning gradient-boosted scorecards with scale_pos_weight and threshold tuning for 1-2% default rates, under RBI model-governance expectations.
• E-commerce and delivery (Flipkart, Myntra, Swiggy, Zomato, Zepto): ranking, ETA and demand models tuned with time-series CV and latency-constrained multi-objective search.
• IT services and analytics firms (TCS, Infosys, Wipro, Accenture, Fractal, Mu Sigma, Quantiphi): client pipelines built on SageMaker AMT/Autopilot, Vertex AI and H2O - "MLOps engineer" and "data scientist" roles explicitly list Optuna/Ray Tune/MLflow.
• Healthcare and agritech (Niramai, Qure.ai, Tarnetar/agri-advisory startups, ICAR and IIT collaborations): small, imbalanced datasets where nested CV and group k-fold matter more than model choice.
• Edge and on-device AI (Jio, Samsung R&D Bengaluru, Qualcomm India, automotive ADAS): hardware-aware NAS and quantisation-aware tuning to fit latency and memory budgets.
• Research and competitions: Kaggle/Analytics Vidhya solutions and GATE DA-level coursework; forecasting load for state discoms and SLDCs with rolling-origin validation.

Interview questions

• Q: Difference between a parameter and a hyperparameter? A: Parameters are learned from data by the optimiser (weights, split thresholds); hyperparameters are set before training and control the model or the training process, and cannot be learned by minimising training loss.
• Q: Why is random search often better than grid search? A: Only a few hyperparameters usually matter; a grid repeats the same values of the important one across many trials, whereas random search gives every trial a fresh value, and its coverage depends on the number of trials, not the dimensionality.
• Q: How does Bayesian optimisation choose the next configuration? A: It fits a cheap surrogate (GP, TPE or random forest) to the trials so far and maximises an acquisition function such as Expected Improvement, trading exploitation of the current best against exploration of uncertain regions.
• Q: What is Hyperband and when would you use it? A: Repeated successive halving with different bracket sizes - many configurations start with a tiny budget and poor ones are killed early; use it for anything trained iteratively, such as deep nets or boosting.
• Q: Why must scaling be inside the cross-validation pipeline? A: Fitting a scaler or imputer on all the data leaks validation-fold statistics into training, so the CV score is optimistically biased; use Pipeline so each fold fits its own.
• Q: You tuned 300 configurations and your validation accuracy is 94% but test is 88%. What happened? A: Overfitting to the validation set through repeated selection; fix with nested CV, a larger or repeated-CV validation estimate, fewer trials or a one-standard-error rule.
• Q: Why sample the learning rate log-uniformly? A: Its effect is multiplicative over orders of magnitude - 0.001 vs 0.01 matters far more than 0.10 vs 0.11 - so uniform sampling would spend almost all trials in the large-value region.
• Q: Which hyperparameter would you tune first in a deep network? A: The learning rate, with its schedule and warm-up; it dominates convergence, and batch size, regularisation and architecture come after.
• Q: How do learning rate and number of boosting rounds interact in XGBoost? A: They trade off inversely - halving eta roughly doubles the trees needed; tune eta with early stopping on a validation set to find n_estimators automatically.
• Q: What is the CASH problem? A: Combined Algorithm Selection and Hyperparameter optimisation - searching simultaneously over which algorithm to use and that algorithm's own conditional hyperparameters, which is what AutoML frameworks solve.
• Q: Where does AutoML fail? A: Problem formulation, label definition, domain feature engineering, leakage in the supplied table, distribution shift, fairness and interpretability constraints, and raw compute cost - it automates search, not thinking.
• Q: Two configurations differ by 0.2% in CV accuracy. Which do you ship? A: Check whether the gap exceeds the standard error across folds; if not, it is noise - pick the simpler, faster or more stable model (the one-standard-error rule).

Common mistakes

• Tuning on the test set, or re-checking test accuracy after every sweep - the reported number is then meaningless.
• Scaling, imputing, selecting features or applying SMOTE on the whole dataset before cross-validation, which leaks.
• Shuffling time-series data in k-fold instead of using rolling-origin validation, giving future-to-past leakage.
• Using unstratified folds on imbalanced data, so some folds contain almost no positive class.
• Sampling learning rate or regularisation strength uniformly instead of log-uniformly, and using a 5-point grid for a continuous knob.
• Declaring a winner on a third-decimal difference without the fold standard deviation, or without fixing seeds.
• Treating AutoML output as a finished model - shipping without checking the pipeline, leakage, latency, explainability and DPDP-compliant data handling.
• Launching huge sweeps with no trial timeout, budget or experiment tracking, so the result cannot be reproduced or costed.

Related topics

Semi-supervised and self-supervised learning · Multimodal AI · Model evaluation metrics and cross-validation · Regularisation and the bias-variance trade-off · Ensemble methods: bagging, boosting and stacking · MLOps, experiment tracking and model deployment