Multiple regression

In short

Multiple regression models one dependent variable Y as a linear function of two or more explanatory variables X₁, X₂, …, X_k, so that the effect of each X can be measured while the others are held constant. It is the workhorse of applied statistics: demand estimation, credit scoring, salary and yield models, and almost every "driver analysis" in an analytics job. Indian exams test it heavily — GATE Statistics (ST), IIT JAM MS, UGC NET, Indian Statistical Service (ISS), ICAI/actuarial papers and B.Sc/B.Stat university papers all ask normal equations, partial and multiple correlation, the ANOVA table, t and F tests, and multicollinearity. In placement rounds for Fractal, Mu Sigma, ZS, Tiger Analytics, CRISIL, banks and insurers, the interview almost always opens with regression assumptions, R² versus adjusted R², and what VIF means.

Explained

The model, its terms and its assumptions

• Population model: Y_i = β₀ + β₁X_{1i} + β₂X_{2i} + … + β_kX_{ki} + u_i, i = 1…n, with k explanatory variables and n observations (need n > k+1).
• β_j is a partial regression coefficient: expected change in Y per unit change in X_j, other X's held fixed. This is what distinguishes it from simple regression, where the effect of X₁ also carries the influence of whatever X₂ it travels with.
• β₀ (intercept): mean Y when all X's are zero — often not meaningful, only a positioning constant.
• "Linear" means linear in the parameters, not in the variables: Y = β₀ + β₁X + β₂X² + u and log Y = β₀ + β₁ log X₁ + u are still linear regression models.
• Classical assumptions (CLRM):
  1. Correct linear specification, no omitted relevant variable.
  2. X's are non-stochastic (or independent of u) and no exact linear relation among them (full column rank of X).
  3. E(u_i) = 0.
  4. Homoscedasticity: Var(u_i) = σ² for all i.
  5. No autocorrelation: Cov(u_i, u_j) = 0, i ≠ j.
  6. For tests and intervals: u_i ~ N(0, σ²) independently.
• Gauss–Markov theorem: under 1–5, OLS estimators are BLUE — best (minimum variance) among linear unbiased estimators. Normality is needed only for the t, F and confidence statements, not for unbiasedness.

Estimation by ordinary least squares

• OLS minimises S = Σ(Y_i − b₀ − b₁X_{1i} − … − b_kX_{ki})². Setting ∂S/∂b_j = 0 gives the normal equations (k+1 equations in k+1 unknowns).
• Derivation outline (two-variable case): differentiate S w.r.t. b₀, b₁, b₂; the first equation gives b₀ = Ȳ − b₁X̄₁ − b₂X̄₂, so the fitted plane passes through (X̄₁, X̄₂, Ȳ). Substituting it reduces the other two to deviation form:
  • Σx₁y = b₁Σx₁² + b₂Σx₁x₂
  • Σx₂y = b₁Σx₁x₂ + b₂Σx₂² (small letters = deviations from means)
• Solve by Cramer's rule or elimination; for k ≥ 3 use the matrix form b = (X′X)⁻¹X′y, where X is n×(k+1) with a column of 1's.
• Residuals e_i = Y_i − Ŷ_i satisfy Σe_i = 0 and Σe_iX_{ji} = 0 — residuals are orthogonal to every regressor, so they carry no linear information left in the X's.
• Error variance estimate: s² = Σe_i²/(n − k − 1); divisor is the residual degrees of freedom, and s² is unbiased for σ².
• Standardised (beta) coefficients: β*_j = b_j × (s_{Xj}/s_Y), unit-free, used to rank the relative importance of drivers measured in rupees, kg and years.

Multiple and partial correlation

• Multiple correlation coefficient R_{Y.12…k} = correlation between Y and Ŷ; it lies in [0, 1] (never negative) and R² is the coefficient of determination.
• Two-variable formula: R²_{Y.12} = (r²_{Y1} + r²_{Y2} − 2r_{Y1}r_{Y2}r_{12})/(1 − r²_{12}).
• Partial correlation r_{Y1.2}: correlation between Y and X₁ after removing the linear effect of X₂ from both — i.e. the correlation of the two sets of residuals. It keeps the sign of the partial effect.
• Relation: 1 − R²_{Y.12} = (1 − r²_{Y1})(1 − r²_{Y2.1}); adding a variable can never reduce R².
• Part (semi-partial) correlation removes X₂ only from X₁, not from Y; its square is the extra R² contributed by X₁.
• R² ≥ max(r²_{Y1}, r²_{Y2}) always; a partial correlation can be larger than the simple one, or of opposite sign (suppressor effect).

Goodness of fit and hypothesis testing

• Partition: TSS = Σ(Y−Ȳ)² = RSS(regression) + ESS(error), with df n−1 = k + (n−k−1).
• R² = RSS/TSS. Adjusted R² = 1 − (1−R²)(n−1)/(n−k−1) penalises extra regressors and can fall or go negative; use it (not R²) to compare models with different numbers of X's.
• ANOVA table / overall F test of H₀: β₁ = β₂ = … = β_k = 0 against "at least one ≠ 0": F = (RSS/k)/(ESS/(n−k−1)) with (k, n−k−1) df. Also F = [R²/k]/[(1−R²)/(n−k−1)].
• Individual t test of H₀: β_j = 0: t = b_j/SE(b_j) with n−k−1 df; SE(b_j) = s√(c_jj), c_jj the j-th diagonal element of (X′X)⁻¹. Confidence interval: b_j ± t_{α/2}·SE(b_j).
• For k = 1 regressor, F = t² exactly. A significant F with all t's insignificant is the classic symptom of multicollinearity.
• Partial (incremental) F test for adding a block of q variables: F = [(ESS_restricted − ESS_full)/q] ÷ [ESS_full/(n−k−1)]. For q = 1 it equals the square of that variable's t.
• Prediction: Ŷ₀ = x₀′b. The interval for the mean response is narrower than the interval for an individual future value (the latter adds s² for the new error term). Extrapolating outside the observed X-region is unsafe.

Problems of the data: multicollinearity, heteroscedasticity, autocorrelation

• Multicollinearity: near-linear dependence among X's. Consequences — OLS stays unbiased but variances inflate, SEs blow up, t's fall, signs flip, coefficients swing with small data changes. Detection: high pairwise r, high R² with insignificant t's, VIF_j = 1/(1 − R²_j) from regressing X_j on the other X's (VIF > 5 is a warning, > 10 commonly treated as serious), tolerance = 1/VIF, condition number of X′X. Remedies: drop or combine variables, use ratios/first differences, add data, principal components or ridge regression. Perfect collinearity makes (X′X)⁻¹ non-existent — no OLS solution.
• Heteroscedasticity: Var(u_i) varies (common with cross-section data on firms, districts, households). OLS remains unbiased but is no longer BLUE and SEs are wrong. Tests: Breusch–Pagan, White, Goldfeld–Quandt, Park, Spearman rank correlation of |e| with X. Remedies: weighted least squares, log transformation, White's heteroscedasticity-robust standard errors.
• Autocorrelation: correlated errors, typical of time-ordered data. Durbin–Watson d = Σ(e_t − e_{t−1})²/Σe_t², 0 ≤ d ≤ 4, d ≈ 2 means no first-order autocorrelation, d near 0 positive, near 4 negative; the table gives inconclusive zones, and DW is invalid when a lagged Y is a regressor (use Breusch–Godfrey/Durbin's h). Remedies: Cochrane–Orcutt, generalised differencing, Newey–West SEs.
• Specification errors: omitting a relevant variable correlated with included ones biases the coefficients (omitted-variable bias); including an irrelevant one only inflates variance. Errors in measuring X cause attenuation bias.

Qualitative regressors, diagnostics and model selection

• Dummy (indicator) variables: code a category as 0/1 — gender, rural/urban, graduate/postgraduate, season. For a variable with m categories use m − 1 dummies plus the intercept; using all m causes the dummy-variable trap (perfect collinearity). The omitted category is the base, and each dummy coefficient is a shift in intercept relative to it.
• Interaction terms (X₁ × D or X₁ × X₂) let a slope differ across groups; a Chow test checks structural stability between two sub-samples or periods.
• Influence diagnostics: standardised/studentised residuals, leverage h_ii (high when x_i is far from the mean vector), Cook's distance (> 1 or > 4/n flagged), DFBETA. An outlier in Y is not the same as a high-leverage point in X.
• Normality check: residual histogram, Q–Q plot, Jarque–Bera/Shapiro–Wilk; residual-versus-fitted plots reveal non-linearity and unequal spread.
• Variable selection: forward, backward and stepwise procedures, all-subsets with Mallows' C_p, AIC, BIC, adjusted R², and cross-validation (PRESS). Stepwise on many candidates inflates the chance of spurious variables — prefer theory-guided selection.
• Extensions examined at B.Sc level: polynomial regression, log-log (elasticities) and semi-log (growth rates) forms, ridge and LASSO for collinear/high-dimensional data, logistic regression when Y is binary (OLS is inappropriate there), GLS when assumptions 4–5 fail.

Key points

• Partial regression coefficient — effect of one X on Y with the other X's held constant; different in general from the simple regression slope.
• Normal equations — k+1 simultaneous equations from least squares; in deviation form the intercept drops out.
• Matrix solution — b = (X′X)⁻¹X′y; needs X to have full column rank.
• Gauss–Markov — OLS is BLUE under linearity, zero-mean, homoscedastic, non-autocorrelated errors; normality is extra, only for inference.
• s² = Σe²/(n−k−1) — unbiased error variance; residual df fall by one for every parameter estimated.
• R² — fraction of variance explained; never falls when a regressor is added, so it cannot compare models of different size.
• Adjusted R² — R² penalised for k; may decrease or be negative; the right comparison statistic.
• Overall F — tests all slopes jointly; individual t tests test one slope at a time, and the two can disagree.
• F = t² when only one restriction is tested.
• Multiple correlation R — always non-negative, equals corr(Y, Ŷ), and R² ≥ any single r²_{Yj}.
• Partial correlation — carries a sign and measures net association; its square is not the share of total variance explained.
• Multicollinearity — inflates variances, not bias; detected by VIF > 5–10; cured by dropping/combining variables or ridge.
• Heteroscedasticity — OLS unbiased but inefficient and SEs wrong; use WLS or robust SEs.
• Durbin–Watson — d ≈ 2 means no first-order autocorrelation; d ≈ 2(1 − r̂).
• Dummy variables — m − 1 for m categories; the dropped one is the reference.
• Cook's distance — identifies observations that materially move the fitted coefficients.
• Prediction interval for one new Y is wider than the confidence interval for the mean Y at the same X.

Formulas and examples

Example 1 — fitting a two-variable regression. Monthly figures for a retail outlet: Y = sales (₹ lakh), X₁ = advertising spend (₹ lakh), X₂ = number of salespersons.

Means: X̄₁ = 3, X̄₂ = 3, Ȳ = 4. Sums of squares and products: Σx₁² = 10, Σx₂² = 10, Σx₁x₂ = 8, Σx₁y = 9, Σx₂y = 9, Σy² = 10. Normal equations: 10b₁ + 8b₂ = 9 and 8b₁ + 10b₂ = 9. Subtracting, 2b₁ − 2b₂ = 0 → b₁ = b₂; then 18b₁ = 9 → b₁ = b₂ = 0.5. b₀ = 4 − 0.5(3) − 0.5(3) = 1. Ŷ = 1 + 0.5X₁ + 0.5X₂ — one extra ₹1 lakh of advertising raises expected sales by ₹0.5 lakh, salespersons held constant.

Example 2 — fit, ANOVA and t tests for the same data. RSS = b₁Σx₁y + b₂Σx₂y = 0.5(9) + 0.5(9) = 9; TSS = Σy² = 10; ESS = 1. R² = 9/10 = 0.90; Adjusted R² = 1 − (0.10)(4/2) = 0.80. F = (9/2)/(1/2) = 9.00 on (2, 2) df. F₀.₀₅(2,2) = 19.00, so not significant at 5% — a high R² on n = 5 proves little. s² = ESS/(n−k−1) = 1/2 = 0.5. With (X′X)⁻¹ (deviation part) = (1/36)[[10, −8], [−8, 10]], c₁₁ = 10/36 = 0.2778. SE(b₁) = √(0.5 × 0.2778) = √0.1389 = 0.373; t = 0.5/0.373 = 1.34 on 2 df (t₀.₀₂₅ = 4.303) → not significant. ANOVA table:

Example 3 — correlations, collinearity and the incremental F. From the same sums: r_{Y1} = 9/√(10×10) = 0.9, r_{Y2} = 0.9, r_{12} = 8/10 = 0.8. Check R²: (0.81 + 0.81 − 2×0.9×0.9×0.8)/(1 − 0.64) = (1.62 − 1.296)/0.36 = 0.324/0.36 = 0.90 ✓ (matches Example 2). Partial correlation r_{Y1.2} = (0.9 − 0.9×0.8)/√[(1 − 0.81)(1 − 0.64)] = 0.18/√(0.19 × 0.36) = 0.18/0.2615 = 0.688 — the net association is far weaker than the simple r of 0.9. VIF = 1/(1 − 0.8²) = 1/0.36 = 2.78 (moderate, acceptable). Dropping X₂: b = Σx₁y/Σx₁² = 0.9, RSS₁ = 0.9 × 9 = 8.1, R² = 0.81. Incremental F for adding X₂ = [(1.9 − 1.0)/1]/[1.0/2] = 0.9/0.5 = 1.80 = t² = 1.34² ✓ — X₂ adds nothing beyond X₁ in this sample. Beta coefficients: s_{X1} = s_Y = √(10/4) = 1.5811, so β*₁ = 0.5 × 1.5811/1.5811 = 0.50; same for X₂ — equal relative importance.

Example 4 — prediction. For X₁ = 6 (₹6 lakh ads) and X₂ = 4 salespersons: Ŷ = 1 + 0.5(6) + 0.5(4) = ₹6 lakh. Note X₁ = 6 lies just outside the observed range (1–5), so this is mild extrapolation; the prediction interval would also be wider than the interval for the mean response at the same point.

Example 5 — dummy variable. Fitted starting-salary model from campus data: Ŷ = 18,000 + 1,200·EXP + 2,500·D, where Y = monthly salary in ₹, EXP = years of experience, D = 1 for postgraduate, 0 for graduate (graduate is the base).

• Graduate, 3 years: 18,000 + 3,600 + 0 = ₹21,600.
• Postgraduate, 3 years: 18,000 + 3,600 + 2,500 = ₹24,100.
• The postgraduate premium is a parallel shift of ₹2,500; to let the experience slope differ by qualification, add the interaction EXP × D. With a three-level qualification variable (graduate/postgraduate/doctorate) use two dummies, not three.

Where it is used

• Analytics and consulting firms (Mu Sigma, Fractal, Tiger Analytics, ZS, EXL, Genpact): marketing-mix models estimating sales from price, promotion, distribution and competitor spend; the output is usually reported as elasticities from a log-log fit.
• Banks, NBFCs and credit bureaus (CIBIL, CRIF High Mark, HDFC, Bajaj Finance): income and loss-given-default models by multiple regression, with logistic regression for default itself; RBI model-risk expectations require documented assumption testing.
• Government statistics and policy (MoSPI, NSO, PLFS and NSS unit-level data, RBI and Ministry of Finance): wage equations on education and experience, crop-yield models on rainfall, fertiliser and irrigation for ICAR/IASRI, inflation and demand forecasting; the ISS examination tests this syllabus directly.
• Insurance and actuarial work (LIC, ICICI Lombard, GIC): pricing models relating claim cost to age, vehicle class, geography and sum assured, with dummies for zones.
• Pharma, FMCG and manufacturing quality functions: response-surface and process models linking yield or defect rate to temperature, pressure and concentration, often with interaction terms.
• Job roles: data analyst, business/statistical analyst, risk modeller, econometrician, market research analyst, research assistant in economics departments and think tanks (NCAER, NIPFP).

Interview questions

• Q: What exactly does a coefficient in multiple regression mean? A: The expected change in Y per unit change in that X with all other X's in the model held constant — a partial effect, not the total effect.
• Q: Why can a simple regression slope differ from the partial slope in multiple regression? A: The simple slope also absorbs the effect of correlated omitted variables; once they enter the model, that borrowed effect is removed and the coefficient can shrink or change sign.
• Q: Difference between R² and adjusted R²? A: R² never falls when a regressor is added, so it is useless for comparing models of different size; adjusted R² penalises each added parameter and can decrease or go negative.
• Q: The F test is significant but no t is. What is happening? A: Classic multicollinearity — the regressors jointly explain Y but are so correlated that each individual coefficient has a large standard error.
• Q: Does multicollinearity bias the estimates? A: No. OLS stays unbiased and consistent; only the variances inflate, making inference unreliable and coefficients unstable.
• Q: How do you detect and treat multicollinearity? A: VIF or tolerance, correlation matrix, condition number; treat by dropping or combining variables, using differences or ratios, collecting more data, or ridge/principal-component regression.
• Q: What does the Gauss–Markov theorem require, and what does normality add? A: Linearity, zero-mean, homoscedastic and non-autocorrelated errors give BLUE; normality is needed only for exact t and F tests and confidence intervals.
• Q: What goes wrong under heteroscedasticity? A: Estimates remain unbiased but inefficient and the standard errors are wrong, so t and F tests mislead; use weighted least squares or robust (White) standard errors.
• Q: How many dummies for a variable with four categories, and why? A: Three, plus the intercept; all four would be perfectly collinear with the intercept — the dummy-variable trap.
• Q: Why is a prediction interval wider than a confidence interval for the mean? A: The mean response has only the uncertainty of the estimated coefficients; an individual value adds the variance σ² of its own error term.
• Q: Y is 0/1 (defaulted or not). Can you use multiple regression? A: Not appropriately — fitted values can fall outside [0,1] and the errors are heteroscedastic and non-normal; use logistic regression.
• Q: How do you decide which variables to keep? A: Theory and business meaning first, then partial F/t tests, adjusted R², AIC/BIC or Mallows' C_p and out-of-sample validation; purely automatic stepwise selection over many candidates produces spurious variables.

Common mistakes

• Dividing by n − 2 instead of n − k − 1 when computing s², standard errors or adjusted R².
• Reading a coefficient as a total effect, or interpreting it causally when the X's are observational and correlated.
• Comparing models by R² rather than adjusted R², AIC or validation error, and then declaring the biggest model best.
• Including all m dummies for m categories, or forgetting that the omitted category is the benchmark for interpretation.
• Treating a high VIF as a bias problem, or mechanically dropping a theoretically essential variable because its t is low.
• Ranking importance by the raw b_j when the X's are in different units — standardised beta coefficients or partial correlations are needed.
• Confusing multiple correlation R (never negative) with partial correlation (signed), and squaring a partial correlation to claim a share of total variance.
• Extrapolating the fitted plane far outside the observed range of the X's, and skipping residual plots and the Durbin–Watson check on time-ordered data.

Related topics

Correlation and regression · Time series analysis · Index numbers · Analysis of variance and design of experiments · Testing of hypotheses · Logistic regression and generalised linear models