Overfitting, bias-variance and regularisation

In short

Overfitting is when a model learns the noise and quirks of the training sample instead of the underlying pattern, so it scores well in training and badly on unseen data; underfitting is the opposite failure, a model too simple to capture the signal. The bias-variance decomposition explains both: expected test error splits into squared bias (systematic error of the model family), variance (sensitivity to which particular training sample was drawn) and irreducible noise. Regularisation — ridge, lasso, elastic net, weight decay, dropout, early stopping, pruning, data augmentation — deliberately adds bias to cut variance, and the amount is tuned by cross-validation, not guessed. This is the most reliably asked machine-learning theory topic in GATE DA (the Data Science and Artificial Intelligence paper, introduced in 2024), in university CBCS/AICTE ML papers, and in technical rounds at TCS, Infosys, Wipro, Accenture, Mu Sigma, Fractal, ZS, Flipkart, Swiggy, Razorpay and bank/NBFC analytics teams, where "your model gave 99% on train and 71% on test — what do you do?" is a standard question.

Explained

Overfitting and underfitting

• Generalisation error is the expected loss on data drawn from the same distribution, not on the rows used to fit. All diagnosis is about the gap between training error and held-out error.
• Overfitting: training error low, validation error much higher, and validation error *rises* as complexity grows. Causes: too many parameters for the sample size, too many features (p close to or above n), too many boosting rounds or tree depth, training too long, leakage of target-related features.
• Underfitting (high bias): training error itself is high and validation error is close to it. Causes: model family too restrictive (linear fit on a curved relation), excessive regularisation, too few features, over-aggressive pruning.
• Capacity / complexity knobs: polynomial degree, tree depth and min_samples_leaf, number of hidden units and layers, k in kNN (small k = high variance), C in SVM (large C = low bias, high variance), λ in ridge/lasso, number of estimators in boosting.
• Diagnosis by learning curve (error vs training-set size): if both curves are high and converge, bias dominates — add capacity or features. If a wide gap persists, variance dominates — add data, add regularisation, reduce features.
• Validation curve (error vs the hyperparameter) gives the U-shape; the minimum of the validation arm is the operating point.
• Double descent is a modern caveat: in very heavily over-parameterised models (deep nets, wide random-feature models) test error can fall again beyond the interpolation threshold, so the classical U-curve is not universal.

Bias-variance decomposition

• Setup: true relation y = f(x) + ε, with E[ε] = 0 and Var(ε) = σ². A learning algorithm trained on a random sample D gives predictor f̂_D.
• Derivation outline (examined in GATE and university theory papers): take expected squared error at a fixed point x₀ over both the noise and the random training set, insert and subtract the average prediction f̄(x₀) = E_D[f̂_D(x₀)], expand the square; the cross term vanishes because E_D[f̂_D − f̄] = 0 and ε is independent of D. Three terms remain.
• Bias = f̄(x₀) − f(x₀): systematic error of the *model family and algorithm*, not of one fitted model.
• Variance = E_D[(f̂_D(x₀) − f̄(x₀))²]: how much the fitted function moves if you re-draw the training sample.
• Noise σ² (Bayes error / irreducible error): cannot be removed by any model; a reported accuracy above this ceiling on a test set is a sign of leakage.
• Trade-off: increasing complexity lowers bias and raises variance. Total error is minimised at intermediate complexity — hence the U-shaped test-error curve.
• The exact additive three-way split holds for squared-error loss. For 0-1 loss the decomposition is not additive in the same clean way (Domingos/Kohavi-Wolpert formulations exist); say so in an exam rather than applying the MSE identity to classification accuracy.
• No Free Lunch: no single learner is best over all possible problems; the choice of bias (inductive bias) is what makes learning possible on a particular problem.
• Ensembles map onto this directly: bagging/random forests mainly cut variance (averaging many low-bias, high-variance trees), boosting mainly cuts bias (sequentially fitting residuals of weak, high-bias learners).

Regularisation: ridge, lasso, elastic net

• Principle: minimise *loss + λ × penalty on the parameters*. λ ≥ 0 is the regularisation strength (sklearn's alpha); in SVM and logistic regression the equivalent knob is C ≈ 1/λ, so large C = weak regularisation.
• Ridge (L2, Tikhonov): penalty λΣwⱼ². Closed-form solution exists and is always invertible for λ > 0, so it fixes multicollinearity and p > n. Shrinks coefficients smoothly towards but never exactly to zero; correlated predictors share the weight.
• Lasso (L1): penalty λΣ|wⱼ|. Non-differentiable at zero, so the optimum sits on an axis for large λ — it drives coefficients exactly to zero and therefore performs feature selection. Solved by coordinate descent / LARS, not by a closed form.
• Why L1 is sparse (geometry): the constraint region Σ|wⱼ| ≤ t is a diamond with corners on the axes; the elliptical contours of the squared-error loss first touch it at a corner. The L2 region is a circle with no corners, so a zero is a measure-zero accident.
• Elastic net: λ(α Σ|wⱼ| + (1−α)Σwⱼ²/2) — sparse like lasso but stable when predictors are highly correlated and when p ≫ n (lasso can select at most n variables).
• Bayesian view: ridge = MAP estimate with a zero-mean Gaussian prior on the weights; lasso = MAP with a Laplace (double-exponential) prior. λ is the ratio of noise variance to prior variance.
• Always standardise features first (zero mean, unit variance) — the penalty is scale-dependent, otherwise a feature measured in rupees is penalised differently from the same feature in lakhs. Do not penalise the intercept.
• Effective degrees of freedom of ridge: df(λ) = Σ dⱼ²/(dⱼ² + λ) where dⱼ are the singular values of the standardised design matrix — a continuous complexity measure between p (λ=0) and 0 (λ→∞).

Other regularisers and practical controls

• Weight decay in neural networks: the gradient step becomes w ← w(1 − ηλ) − η ∂L/∂w, i.e. L2 implemented as multiplicative shrinkage each update. (With Adam, decoupled weight decay / AdamW is the correct form.)
• Dropout: randomly zero a fraction p of unit activations during training (scaling the rest), use the full network at test time; acts like training an exponential ensemble of thinned networks.
• Early stopping: stop when validation loss stops improving (patience). For gradient descent on linear models it is provably close to L2 shrinkage.
• Batch normalisation / layer normalisation have a mild regularising side-effect through minibatch noise, but they are not substitutes for explicit regularisation.
• Data augmentation (image flips/crops, synonym replacement, SMOTE for minority classes) raises the effective sample size and is often the strongest variance reducer available.
• Tree controls: max_depth, min_samples_leaf, min_impurity_decrease, cost-complexity pruning (ccp_alpha, the α in R_α(T) = R(T) + α|T|); in XGBoost/LightGBM: lambda (L2), alpha (L1), gamma (min split loss), subsample, colsample_bytree, early stopping rounds.
• kNN and SVM: increase k; for SVM reduce C and lower the RBF gamma (a wider kernel) to smooth the boundary.
• Noise injection, label smoothing, max-norm constraints, parameter sharing (CNN weights) are all regularisers in the same sense: they restrict or bias the hypothesis space.
• Get more data when feasible — variance falls roughly as 1/n; bias does not fall at all, so more data never fixes underfitting.

Choosing the right complexity: validation protocol

1. Split first, then look. Train / validation / test, typically 60:20:20, or k-fold CV on train with a locked-away test set used once.
2. k-fold CV: split into k folds (k = 5 or 10), train on k−1, score on the held-out fold, average. Stratified k-fold for classification keeps class proportions.
3. LOOCV (k = n): nearly unbiased but high variance and expensive; for linear models it is cheap via the hat-matrix shortcut.
4. Grid search / random search / Bayesian optimisation over λ (usually on a log grid such as 10⁻⁴ … 10³), scored by CV mean, with the standard error reported.
5. One-standard-error rule: among models within one SE of the best CV score, pick the simplest (largest λ) — a classic exam point and good practice.
6. Nested CV when you both tune and estimate performance; otherwise the reported score is optimistically biased by the tuning.
7. Respect structure: time series needs forward-chaining splits (TimeSeriesSplit), never random shuffling; grouped data (multiple rows per customer) needs GroupKFold to prevent leakage.
8. Fit preprocessing inside the fold — scalers, imputers, PCA, SMOTE, target encoding must be fitted on the training part only. Use a Pipeline.

Leakage, drift and the Indian compliance angle

• Data leakage is the commonest cause of a model that looks perfect in the lab and fails in production: scaling before splitting, including a post-outcome field (e.g. "collection agency assigned" when predicting default), duplicate rows across splits, or tuning on the test set.
• Distribution shift / concept drift: a well-generalising model still decays as behaviour changes; monitor PSI/KS on inputs and scores, retrain on a schedule.
• Reproducibility: fix random seeds, log λ and fold indices, version the data — the same discipline exam vivas and employers both check.
• Indian regulatory practice: RBI expects credit models and default-risk models to be validated and governed (model documentation, back-testing, periodic validation), and its digital-lending framework requires lenders to be accountable for automated decisioning, so an unexplained overfitted model is a compliance problem, not only a technical one. Sparse, regularised linear/logistic models and scorecards remain popular in Indian BFSI partly for this reason. The Digital Personal Data Protection Act, 2023 governs personal-data use in training (notice, purpose limitation); its rules and phased commencement have been notified progressively, so check the current status of specific provisions before quoting dates.

Key points

• Overfitting - low training error, much higher test error; model has learned noise.
• Underfitting - high training *and* test error; model family too simple or too constrained.
• Generalisation gap - validation error minus training error; the quantity regularisation targets.
• Bias - error from wrong assumptions of the model family, averaged over training sets.
• Variance - error from sensitivity to the particular training sample drawn.
• Irreducible noise σ² - Bayes error floor; no model beats it legitimately.
• Decomposition - E[(y − f̂)²] = bias² + variance + σ², exact for squared-error loss only.
• Complexity U-curve - test error falls then rises; the minimum is the operating point.
• Ridge (L2) - shrinks all coefficients, keeps them all, handles multicollinearity, closed form.
• Lasso (L1) - sets some coefficients exactly to zero, so it selects features; no closed form.
• Elastic net - L1 + L2; sparse and stable with correlated predictors and p ≫ n.
• λ (alpha) - regularisation strength; λ→0 gives OLS, λ→∞ gives the null model. In SVM/logistic, C ≈ 1/λ.
• Standardise before penalising and never penalise the intercept.
• Bagging cuts variance, boosting cuts bias - the ensemble mapping of the trade-off.
• Early stopping, dropout, weight decay, augmentation, pruning - regularisers outside the loss penalty.
• k-fold CV - the standard λ selector; stratified for classes, forward-chaining for time series.
• One-standard-error rule - prefer the simplest model within 1 SE of the best CV score.
• Leakage - preprocessing or features that peek at the target/test; fakes good generalisation.

Formulas and examples

Example 1 — bias-variance decomposition from resampled models. Five models trained on five bootstrap samples predict at x₀: 8, 9, 11, 12, 15. True f(x₀) = 10, σ² = 1.

• Mean prediction = (8+9+11+12+15)/5 = 55/5 = 11.0; Bias = 11.0 − 10 = 1.0 → bias² = 1.0.
• Deviations: −3, −2, 0, +1, +4 → squares 9, 4, 0, 1, 16; sum = 30 → Variance = 30/5 = 6.0.
• Expected MSE = 1.0 + 6.0 + 1.0 = 8.0.

A simpler (more regularised) model on the same five samples predicts 7, 7.5, 8, 8, 8.5.

• Mean = 39/5 = 7.8; Bias = −2.2 → bias² = 4.84.
• Deviations: −0.8, −0.3, +0.2, +0.2, +0.7 → 0.64 + 0.09 + 0.04 + 0.04 + 0.49 = 1.30 → Variance = 0.26.
• Expected MSE = 4.84 + 0.26 + 1.00 = 6.10.

The biased, stabler model wins by 1.90 — that is the whole argument for regularisation.

Example 2 — ridge and lasso by hand (one feature, no intercept). Data (x, y): (1, 2), (2, 3), (3, 7). Σx² = 1 + 4 + 9 = 14; Σxy = 2 + 6 + 21 = 29.

• OLS: ŵ = 29/14 = 2.0714.
• Ridge λ = 6: ŵ = 29/(14 + 6) = 29/20 = 1.45 (30% shrinkage).
• Ridge λ = 14: ŵ = 29/28 = 1.0357 — exactly half of OLS, since λ = Σx².
• Lasso λ = 10: ŵ = (29 − 5)/14 = 24/14 = 1.7143.
• Lasso λ = 58: ŵ = (29 − 29)/14 = 0 exactly — the feature is dropped. Threshold is λ = 2|Σxy| = 58; ridge would never reach zero for any finite λ.

Example 3 — choosing polynomial degree from a validation curve.

Degree 6 minimises validation RMSE at 3.0; degree 12's train RMSE of 0.1 is memorisation, and its validation error is 3.7× worse.

Example 4 — selecting λ by 5-fold CV with the one-SE rule. Fold MSEs:

Best mean is λ = 1 at 0.330. For λ = 1 the deviations from 0.330 are 0.00, 0.02, −0.02, 0.03, −0.03 → sum of squares 0.0026, sample variance 0.0026/4 = 0.00065, s = 0.0255, SE = 0.0255/√5 ≈ 0.0114. One-SE band = 0.330 + 0.011 = 0.341; λ = 0.1 (0.370) lies outside it, so λ = 1 stands.

Example 5 — ridge shrinkage and effective df. Standardised design with singular values d = (√10, 2, 1), i.e. d² = 10, 4, 1, and λ = 2.

• Shrinkage factors: 10/12 = 0.833, 4/6 = 0.667, 1/3 = 0.333.
• df(2) = 0.833 + 0.667 + 0.333 = 1.833 against p = 3 for OLS — the low-variance directions are damped hardest, which is exactly how ridge stabilises collinear data.

Example 6 — one weight-decay step. w = 0.80, η = 0.1, λ = 0.5, ∂L/∂w = 0.2: w ← 0.80 × (1 − 0.1×0.5) − 0.1 × 0.2 = 0.80 × 0.95 − 0.02 = 0.76 − 0.02 = 0.74.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV
import numpy as np

alphas = np.logspace(-4, 3, 50)          # lambda grid
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=alphas, cv=5))
lasso = make_pipeline(StandardScaler(), LassoCV(alphas=alphas, cv=5, max_iter=10000))
enet  = make_pipeline(StandardScaler(),
                      ElasticNetCV(l1_ratio=[.1,.5,.7,.9,.95,1], alphas=alphas, cv=5))
# scaler is fitted inside each CV fold because it is in the pipeline - no leakage
ridge.fit(X_train, y_train)
print(ridge[-1].alpha_, ridge.score(X_test, y_test))

Where it is used

• BFSI credit and fraud models — HDFC, ICICI, Axis, Bajaj Finance, CRED, Razorpay: regularised logistic regression and L1/L2-penalised gradient boosting for scorecards that must stay stable and explainable under RBI model-validation expectations.
• E-commerce and consumer internet — Flipkart, Myntra, Swiggy, Zomato, Meesho: demand forecasting, ranking and churn models where early stopping and subsampling in XGBoost/LightGBM are the day-to-day regularisation levers.
• Analytics and consulting firms — Mu Sigma, Fractal, ZS Associates, Tiger Analytics, Quantiphi: cross-validation protocol and overfitting control are core interview screens for associate data-scientist roles.
• Healthcare and genomics — p ≫ n settings (gene expression, radiology features) where lasso and elastic net are standard for selecting a few informative predictors from thousands.
• Deep learning teams — dropout, weight decay/AdamW, augmentation and early stopping in computer-vision and NLP stacks at Indian product and GCC teams (Samsung R&D, Qualcomm, Nvidia, Sarvam, Krutrim).
• Roles — data scientist, ML engineer, risk/credit modeller, quantitative analyst, MLOps engineer (drift monitoring and retraining triggers).

Interview questions

• Q: Define overfitting and give two symptoms? A: The model fits noise in the training sample rather than the underlying pattern; symptoms are a large train-to-validation error gap and validation error that worsens as complexity or training time increases.
• Q: Write the bias-variance decomposition and say when it holds exactly? A: Expected test error = bias² + variance + irreducible noise σ²; the clean additive form holds for squared-error loss, not for 0-1 loss.
• Q: Will collecting more data fix underfitting? A: No — more data reduces variance roughly as 1/n but leaves bias unchanged; underfitting needs more capacity, better features or less regularisation.
• Q: Why does lasso give exact zeros and ridge not? A: The L1 constraint region is a diamond with corners on the axes, so the loss contours first touch it at a corner; the L2 ball is smooth, so the optimum almost never lands exactly on an axis.
• Q: When would you prefer elastic net over lasso? A: With highly correlated predictors or p ≫ n — lasso picks one of a correlated group arbitrarily and can select at most n features, while the L2 part keeps correlated predictors together and stabilises the path.
• Q: What does λ → ∞ do in ridge regression? A: All penalised coefficients shrink towards zero, so the model collapses to predicting the intercept (the mean of y) — maximum bias, minimum variance.
• Q: Why must features be standardised before ridge or lasso? A: The penalty is scale-dependent, so a feature in different units would be shrunk by a different effective amount; the intercept is left unpenalised.
• Q: How is early stopping a regulariser? A: It limits how far the optimiser travels from the small-norm initialisation, which for gradient descent on linear models is approximately equivalent to an L2 penalty.
• Q: Your model gets 0.99 AUC on validation and 0.68 in production. First hypothesis? A: Data leakage — a feature available only after the outcome, or preprocessing/splitting done on the full dataset — followed by distribution shift between the training period and production.
• Q: What is the one-standard-error rule? A: Among all models whose CV error is within one standard error of the minimum, choose the simplest (most regularised) one, for a more robust and parsimonious model.
• Q: Bagging vs boosting in bias-variance terms? A: Bagging averages many low-bias, high-variance models and mainly reduces variance; boosting sequentially fits residuals of weak high-bias learners and mainly reduces bias (so it can overfit without early stopping).
• Q: Is a deep network with far more parameters than data points always overfitted? A: Not necessarily — implicit regularisation from SGD plus dropout, weight decay and augmentation can keep test error low, and the double-descent phenomenon shows test error can fall again beyond the interpolation point.
• Q: How do you regularise a decision tree, a kNN and an SVM? A: Tree: limit depth, raise min_samples_leaf, prune with ccp_alpha; kNN: increase k; SVM: reduce C and lower the RBF gamma (wider kernel).

Common mistakes

• Treating the test set as a validation set — tuning λ on it, then quoting its score as the generalisation estimate (use nested CV or a locked test set used once).
• Fitting the scaler, imputer, PCA or SMOTE on the whole dataset before splitting — leakage that inflates CV scores.
• Applying the bias² + variance + σ² identity to classification accuracy, where the decomposition is not additive in that form.
• Confusing the direction of the knob: larger λ means *more* regularisation, but larger C in SVM/logistic means *less*.
• Treating bias and variance as properties of one fitted model instead of expectations over training samples.
• Forgetting to standardise features, or penalising the intercept, in ridge/lasso — then concluding "ridge did not help".
• Shuffling time-series rows into random folds, so the model is validated on the past using the future.
• Reading a single train-test split's gap as proof of overfitting when the test set is tiny — report CV mean with its standard error.

Related topics

Machine learning workflow · Model evaluation metrics · Linear and logistic regression · Ensemble methods and gradient boosting · Feature selection and model interpretability · Dimensionality reduction and PCA