Linear and logistic regression

In short

Linear regression fits a straight-line (or hyperplane) relationship between numeric predictors and a continuous target by minimising squared error; logistic regression models the probability of a class by passing a linear score through the sigmoid and fitting by maximum likelihood. Both are *generalised linear models* — linear in the parameters — and both are the baseline every data-science team builds before anything fancier. They are the most heavily examined topics in ML papers under AICTE/CBCS curricula and in GATE DA (Data Science and Artificial Intelligence, introduced 2024), and the most asked in Indian analytics interviews (Mu Sigma, Fractal, Tiger Analytics, ZS Associates, TCS/Infosys/Wipro data units, bank and NBFC risk teams). Interviewers favour them because the maths is closed-form enough to probe and the coefficients are interpretable enough to defend to a regulator.

Explained

Simple and multiple linear regression

• Model: y = β₀ + β₁x₁ + β₂x₂ + … + β_p x_p + ε, where ε is random error with E(ε)=0. β₀ is the intercept, β_j the partial slope — the change in y per unit change in x_j, all other predictors held constant.
• Least squares (OLS): choose β to minimise the residual sum of squares SSE = Σ(yᵢ − ŷᵢ)². Residual eᵢ = yᵢ − ŷᵢ.
• Derivation outline (simple case): set ∂SSE/∂β₀ = 0 and ∂SSE/∂β₁ = 0 → the two normal equations Σy = nβ₀ + β₁Σx and Σxy = β₀Σx + β₁Σx². Solving gives β₁ = Sxy/Sxx and β₀ = ȳ − β₁x̄. The fitted line always passes through (x̄, ȳ) and Σeᵢ = 0.
• Matrix form: SSE = (y − Xβ)ᵀ(y − Xβ); setting the gradient −2Xᵀ(y − Xβ) = 0 gives XᵀXβ = Xᵀy, so β̂ = (XᵀX)⁻¹Xᵀy. X is n×(p+1) with a column of 1s for the intercept.
• Computation: closed form needs XᵀX invertible (O(p³)); for large p or n use gradient descent (batch, stochastic, mini-batch) or QR/SVD decomposition, which is numerically safer than inverting XᵀX.
• Non-linearity without leaving the model: polynomial terms (x², x³), interactions (x₁x₂) and log/√ transforms keep the model *linear in parameters* and are still fitted by OLS.
• Interpreting transforms: log-log → β is an elasticity (% change in y per % change in x); semi-log (log y) → 100β ≈ % change in y per unit of x.
• Categorical predictors: a k-level category becomes k−1 dummy variables; including all k with an intercept causes the dummy variable trap (perfect multicollinearity, XᵀX singular).

Assumptions, diagnostics and inference

• LINE + Gauss–Markov: (1) Linearity in parameters; (2) Independent errors (no autocorrelation); (3) Normal errors — needed only for t/F tests and confidence intervals, not for OLS itself; (4) Equal variance (homoscedasticity); plus no perfect multicollinearity and predictors uncorrelated with ε (exogeneity).
• Gauss–Markov theorem: under these, OLS is BLUE — the Best (minimum-variance) Linear Unbiased Estimator.
• Residual plots: residuals vs fitted should be a structureless band. A funnel = heteroscedasticity (fix by log transform or weighted least squares / robust "sandwich" standard errors); a curve = missing non-linear term; Q-Q plot checks normality.
• Autocorrelation: Durbin–Watson statistic ≈ 2 means none; near 0 positive, near 4 negative. Common in time-series data (sales, demand).
• Multicollinearity: predictors correlated with each other. Coefficients become unstable with inflated standard errors (signs may flip) though predictions stay usable. Measured by VIF_j = 1/(1 − R²_j); VIF > 5 is a warning, > 10 usually treated as serious. Remedies: drop/combine variables, PCA, ridge.
• Outliers and influence: large standardised residual = outlier in y; high leverage = unusual x; Cook's distance > ~1 (or 4/n as a screen) flags an influential point that is bending the line.
• Significance tests: t = β̂_j / SE(β̂_j) with n − p − 1 degrees of freedom tests H₀: β_j = 0; the overall F-test = MSR/MSE tests whether all slopes are jointly zero.
• Goodness of fit: R² = SSR/SST = 1 − SSE/SST is the fraction of variance explained; it never falls when a predictor is added, so use adjusted R² for model comparison, or AIC/BIC.

Logistic regression — model and fitting

• Why not OLS on a 0/1 target: a line gives probabilities below 0 and above 1, the errors are heteroscedastic and non-normal, and the fit is dragged by extreme points.
• Model: p = P(y=1 | x) = σ(z) = 1/(1 + e^−z) with z = β₀ + Σβ_j x_j. Equivalently logit(p) = ln[p/(1−p)] = z — the model is linear in the log-odds, not in p.
• Sigmoid properties: S-shaped, range (0,1), σ(0) = 0.5, σ′(z) = σ(z)[1 − σ(z)] (maximum 0.25 at z = 0) — this derivative makes the gradients clean.
• Coefficient meaning: e^β_j is the odds ratio — the multiplicative change in the odds of y=1 per unit increase in x_j. β_j > 0 raises the probability; the effect on p itself is non-linear (steepest near p = 0.5).
• Fitting by maximum likelihood: likelihood L = Π pᵢ^yᵢ (1−pᵢ)^1−yᵢ; maximising its log is the same as minimising log loss / binary cross-entropy, which is convex, so there is a single global optimum but no closed form.
• Gradient (elegant result): ∂J/∂β = (1/n) Xᵀ(p − y) — identical in form to linear regression, with p replacing ŷ. Solved by gradient descent, Newton–Raphson, or IRLS (iteratively reweighted least squares, weights w = p(1−p)); libraries use quasi-Newton solvers such as lbfgs, or Newton/coordinate-descent solvers such as newton-cg and liblinear.
• Decision boundary: predict class 1 when p ≥ threshold (default 0.5, i.e. z ≥ 0). The boundary is a linear hyperplane in feature space — a logistic model cannot separate XOR-type data without engineered features.
• Threshold is a business choice, not a constant: in credit approval or disease screening the cost of a false negative differs from a false positive, so the cut-off is tuned on a validation set (ROC/precision-recall).
• Perfect separation (a feature that splits classes exactly) drives coefficients to ±∞ and the solver fails to converge — fixed by regularisation (sklearn applies L2 by default, strength 1/C) or Firth's penalised likelihood.
• Variants: multinomial logistic / softmax for k > 2 unordered classes (p_k = e^z_k/Σe^z_j), one-vs-rest as an alternative, ordinal logit for ordered outcomes (grades, ratings), probit when the normal CDF is used instead of the sigmoid.
• Fit measures: no true R². Use McFadden pseudo-R² = 1 − (lnL_model/lnL_null), the likelihood-ratio test (−2ΔlnL ~ χ²), Wald tests per coefficient, Hosmer–Lemeshow calibration test, and AUC.

Regularisation, scaling and class balance

• Ridge (L2): minimise SSE + λΣβ_j²; shrinks coefficients smoothly, handles multicollinearity, never gives exact zeros; introduces bias but cuts variance.
• Lasso (L1): SSE + λΣ|β_j|; drives some coefficients exactly to zero, so it also selects features. Elastic net mixes both.
• The intercept is not penalised, and predictors must be standardised first, otherwise the penalty depends on units (₹ vs ₹ lakh). Scaling also makes gradient descent converge far faster.
• λ (or C = 1/λ) is chosen by k-fold cross-validation, not by looking at training error.
• Imbalanced classes (fraud ~0.1%): accuracy is useless; use class weights, resampling, a tuned threshold, and PR-AUC.

Comparison, strengths and limits

• Shared strengths: fast to train and score, few hyperparameters, interpretable coefficients, work on small datasets, give calibrated-ish probabilities (logistic) and standard errors, easy to audit.
• Shared limits: assume a linear (or log-odds-linear) relationship, sensitive to outliers and to multicollinearity, need manual feature engineering for interactions, and extrapolate badly outside the training range.
• Against tree ensembles: gradient boosting usually wins on raw accuracy with tabular data, but regression wins on interpretability, training cost and regulatory defensibility — which is why Indian banks and NBFCs still ship logistic scorecards.
• Indian practice and compliance: credit scorecards built on logistic regression must be explainable under RBI's digital-lending and model-governance expectations; personal data used as features falls under the Digital Personal Data Protection Act, 2023 (operational rules were notified in 2025 with phased timelines — check the current notification before quoting dates). Insurance pricing models come under IRDAI oversight. Features that proxy for caste, religion or gender are a fairness and legal risk and are normally excluded.

Key points

• Linear regression — predicts a continuous y; loss = squared error; closed-form solution exists.
• Logistic regression — predicts P(class); loss = log loss; no closed form, solved iteratively.
• Normal equation — β̂ = (XᵀX)⁻¹Xᵀy; fails if XᵀX is singular (collinear or p > n).
• Logit link — ln[p/(1−p)] = Xβ; linear in log-odds, S-shaped in p.
• Odds ratio — e^β_j; β_j = 0.69 means odds roughly double.
• R² — variance explained; adjusted R² penalises extra predictors; R² is not a measure of causality or correctness.
• Gauss–Markov / BLUE — OLS has minimum variance among linear unbiased estimators under the standard assumptions.
• Normality of errors — needed for inference (t, F, CIs), not for the OLS estimate itself.
• Heteroscedasticity — unequal error variance; estimates stay unbiased, standard errors become wrong.
• VIF — 1/(1−R²_j); > 10 signals serious multicollinearity.
• Dummy variable trap — drop one level of every categorical variable when an intercept is present.
• Sigmoid derivative — σ′ = σ(1−σ); used in the gradient and in backpropagation.
• Convexity — both losses are convex, so there are no local minima; any good optimiser finds the global optimum.
• Threshold — 0.5 is only a default; move it to trade recall against precision.
• Feature scaling — mandatory for gradient descent and regularisation, irrelevant for the closed-form OLS estimate.
• Regularisation — ridge shrinks, lasso selects; both trade a little bias for much less variance.
• Perfect separation — in logistic regression, coefficients blow up; regularise.
• Multicollinearity vs prediction — hurts coefficient interpretation far more than predictive accuracy.

Formulas and examples

Example 1 — Fit a simple regression and test the slope. Data: x = 1,2,3,4,5 (hours of practice tests per week); y = 2,4,5,4,5 (marks scored, scale 0–10).

• n = 5, Σx = 15, Σy = 20, x̄ = 3, ȳ = 4, Σxy = 66, Σx² = 55.
• Sxx = 55 − 15²/5 = 10; Sxy = 66 − (15×20)/5 = 6.
• β₁ = 6/10 = 0.6; β₀ = 4 − 0.6×3 = 2.2 → ŷ = 2.2 + 0.6x.
• Fitted: 2.8, 3.4, 4.0, 4.6, 5.2. Residuals: −0.8, 0.6, 1.0, −0.6, −0.2 (sum = 0 ✓).
• SSE = 0.64+0.36+1.00+0.36+0.04 = 2.4; SST = 4+0+1+0+1 = 6; SSR = 3.6.
• R² = 3.6/6 = 0.60; Adjusted R² = 1 − 0.40×(4/3) = 0.467; RMSE = √(2.4/5) = 0.693.
• s² = 2.4/3 = 0.8, s = 0.894; SE(β₁) = 0.894/√10 = 0.283; t = 0.6/0.283 = 2.12.
• t(0.025, df=3) = 3.182 > 2.12 → slope not significant at 5% with only five points: a decent R² does not guarantee significance in a small sample.

Example 2 — Two steps of batch gradient descent (same data), α = 0.1, start β = (0, 0).

• Step 1: ŷ = 0 everywhere. (1/n)Σ(ŷ−y) = −20/5 = −4; (1/n)Σ(ŷ−y)x = −66/5 = −13.2.
• β₀ = 0 − 0.1(−4) = 0.4; β₁ = 0 − 0.1(−13.2) = 1.32.
• Step 2: ŷ = 1.72, 3.04, 4.36, 5.68, 7.00; errors = −0.28, −0.96, −0.64, +1.68, +2.00. Σe = 1.8 → grad₀ = 0.36; Σe·x = 12.6 → grad₁ = 2.52.
• β₀ = 0.4 − 0.036 = 0.364; β₁ = 1.32 − 0.252 = 1.068.
• β₁ overshot 0.6 on the first step and is now coming back — the classic sign that α is too large / features are unscaled. Scaling x or using α = 0.02 converges monotonically to (2.2, 0.6).

Example 3 — Logistic regression: probability and odds ratio. Placement model: z = −6 + 0.6·CGPA + 1.2·Internship (Internship = 1 if completed).

• Student A, CGPA 8.0, internship = 1 → z = −6 + 4.8 + 1.2 = 0 → p = 1/(1+e⁰) = 0.500 (exactly on the boundary).
• Student B, CGPA 8.0, internship = 0 → z = −1.2 → p = 1/(1 + 3.3201) = 0.2315; odds = 0.2315/0.7685 = 0.301.
• Student C, CGPA 9.0, internship = 1 → z = +0.6 → p = 1/(1 + 0.5488) = 0.646.
• Odds ratio for an internship = e^1.2 = 3.32 → an internship multiplies the odds of placement by about 3.3 at any CGPA; per CGPA point the odds multiply by e^0.6 = 1.82.
• Note the non-linearity: the same +1 CGPA raised p by 0.146 for an internship holder (0.500→0.646) but by only about 0.12 for B (0.2315→0.3543, since z = −0.6 → p = 1/2.8221).

Example 4 — Threshold metrics and log loss. 100 students, threshold 0.5: TP = 35, FP = 15, FN = 10, TN = 40.

Log loss on four held-out cases — (y=1, p=0.9), (y=0, p=0.2), (y=1, p=0.6), (y=0, p=0.7): −[ln0.9 + ln0.8 + ln0.6 + ln0.3]/4 = (0.1054 + 0.2231 + 0.5108 + 1.2040)/4 = 0.511 nats. The single confident wrong call (p = 0.7 for a y = 0) contributes 59% of the loss — log loss punishes confident errors hardest.

Example 5 — Ridge shrinkage and multicollinearity. Using Example 1's centred data (Sxx = 10, Sxy = 6): λ = 0 → β₁ = 0.600; λ = 5 → 6/15 = 0.400; λ = 20 → 6/30 = 0.200. Coefficients shrink towards zero but never reach it. If a second predictor x₂ = 2x₁ were added, XᵀX would be singular and OLS would have no unique solution, while ridge still returns a stable (split) pair of coefficients.

from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np, statsmodels.api as sm

X = np.array([[1],[2],[3],[4],[5]]); y = np.array([2,4,5,4,5])
lin = LinearRegression().fit(X, y)
print(lin.intercept_, lin.coef_, lin.score(X, y))   # 2.2  [0.6]  0.6

print(sm.OLS(y, sm.add_constant(X)).fit().summary())  # t, p, F, CIs

# logistic with scaling + L2 (C = 1/lambda); predict_proba, not predict, for probabilities
clf = make_pipeline(StandardScaler(),
                    LogisticRegression(C=1.0, class_weight='balanced', max_iter=1000))

Where it is used

• Banking and NBFC credit risk: logistic-regression scorecards for probability of default, application and behaviour scoring, and PD models under IFRS-9/ECL style provisioning — used because RBI-facing documentation demands explainable coefficients (HDFC, ICICI, Bajaj Finance, bureau models at TransUnion CIBIL/Experian).
• Insurance and healthtech: IRDAI-regulated pricing and claim-propensity models; linear regression for claim amount, logistic for claim/no-claim and lapse prediction (Policybazaar, LIC/ICICI Lombard analytics teams).
• E-commerce and consumer internet: demand and price-elasticity regressions, delivery-time ETA baselines, logistic models for churn, conversion, RTO (return-to-origin) and fraud screening at Flipkart, Myntra, Swiggy, Zomato, Paytm.
• Marketing and pharma analytics: marketing-mix modelling (log-log regressions on spend to measure ROI) and physician-prescribing propensity at Mu Sigma, Fractal, ZS Associates, Tiger Analytics — regression is the core skill these firms test.
• Infrastructure and energy: load/demand forecasting for state discoms, concrete-strength and material-property regressions, traffic and ridership models for metro DPRs.
• Job roles and academics: data analyst, business analyst, risk analyst, data scientist, MLE; and in research — hedonic house-price models (₹/sq ft against area, floor, locality dummies), NSSO/NFHS survey analysis using logit for binary outcomes.

Interview questions

• Q: Why can't we use linear regression for a binary outcome? A: It predicts values outside [0,1], violates homoscedasticity and normality of errors, and the fitted line is pulled by extreme x values; the logit link fixes all three.
• Q: Is logistic regression a linear model? A: Yes — it is linear in the parameters and its decision boundary is a hyperplane; only the mapping from the linear score to probability is non-linear.
• Q: What does the coefficient 0.8 mean in a logistic model? A: A one-unit rise in that feature multiplies the odds of the positive class by e^0.8 ≈ 2.23, holding other features constant.
• Q: Why maximum likelihood for logistic and least squares for linear? A: Minimising squared error is the MLE under normal errors; for a Bernoulli target the MLE is the cross-entropy minimiser, and squared error there is non-convex in the parameters and poorly scaled.
• Q: Can R² be negative? A: Not for OLS on its training data, but on a test set or for a model without an intercept it can be, meaning the model predicts worse than the mean.
• Q: What breaks when predictors are highly correlated? A: XᵀX becomes near-singular, so coefficient variances inflate and signs can flip; prediction accuracy is largely unaffected. Detect with VIF, treat with dropping/combining features, PCA or ridge.
• Q: Does feature scaling change OLS coefficients or predictions? A: It rescales coefficients but leaves predictions and R² identical; it matters for gradient descent speed and for regularisation, which otherwise penalises large-unit features unfairly.
• Q: How do you handle an imbalanced target in logistic regression? A: Use class weights or resampling, judge with PR-AUC/recall rather than accuracy, and tune the threshold to the cost of each error type instead of leaving it at 0.5.
• Q: Your logistic solver says "failed to converge" — why? A: Usually perfect or near-perfect separation (often a leaked feature), unscaled features, or too few iterations; add L2 regularisation, scale, raise max_iter, and hunt for leakage.
• Q: When would you prefer logistic regression over gradient boosting? A: When the dataset is small, the relationship is roughly log-odds-linear, calibrated probabilities are needed, or the model must be explained to a regulator or credit committee.
• Q: What is the difference between R² and adjusted R²? A: R² never decreases when a predictor is added; adjusted R² subtracts a penalty for the number of predictors and can fall, so only it is valid for comparing models of different size.
• Q: Why is the log-loss surface for logistic regression easier to optimise? A: It is convex in the parameters, so there is a single global minimum and any gradient-based solver converges to it, given a sensible learning rate.

Common mistakes

• Reading a significant slope as proof of causation, or reading a high R² as proof the model is correct — omitted variables and confounders break both claims.
• Interpreting a logistic coefficient as a change in probability instead of in log-odds, or quoting e^β as a *risk* ratio when it is an *odds* ratio.
• Keeping all k dummies for a k-level category along with the intercept (dummy variable trap) and then wondering why the solver reports singularity or drops a term.
• Forgetting to standardise features before ridge/lasso or gradient descent, and using the same learning rate on features measured in ₹ and in years.
• Using predict instead of predict_proba and then trying to plot an ROC curve, or reporting accuracy on a 99:1 imbalanced dataset.
• Skipping residual diagnostics: fitting a straight line to curved data, or treating heteroscedastic standard errors as valid t-tests.
• Tuning λ or the threshold on the test set, and scaling or imputing before the train–test split — both leak information and inflate the reported score.

Related topics

Overfitting, bias-variance and regularisation · Model evaluation metrics · Feature selection and model interpretability · Machine learning workflow · Support vector machines and k-nearest neighbours · Dimensionality reduction and PCA