Multivariate analysis

In short

Multivariate analysis studies several correlated response variables together instead of one at a time, using vectors, mean vectors and covariance matrices in place of scalars. Its core tools are the multivariate normal distribution, Hotelling's T², MANOVA, principal component analysis, factor analysis, discriminant analysis, canonical correlation and cluster analysis. It is a compulsory paper in B.Sc. Statistics under UGC CBCS and appears in GATE Statistics (ST), ISI/CMI, IIT JAM MS, UGC-NET Statistics and the ISS examination. In Indian industry it is the statistical backbone of credit scoring at banks and NBFCs, customer segmentation in telecom and FMCG, psychometric scale validation, NSO/NSSO survey analysis and analytics roles at Mu Sigma, ZS, Fractal, TCS and Deloitte.

Explained

Random vectors, mean vector and dispersion matrix

• A random vector X = (X₁, …, X_p)′ has mean vector μ = E(X) and dispersion (variance–covariance) matrix Σ = E[(X−μ)(X−μ)′], with σ_ij = Cov(X_i, X_j) on the off-diagonal and variances on the diagonal.
• Σ is symmetric and non-negative definite; it is positive definite unless some linear combination a′X is degenerate (then the distribution is singular and |Σ| = 0).
• Correlation matrix R = D^−1/2 Σ D^−1/2, where D = diag(σ₁₁, …, σ_pp). R has 1's on the diagonal and is scale-free.
• Generalised variance = |Σ|; total variance = tr(Σ). Both summarise spread by a single number; |Σ| → 0 signals near-collinearity.
• Linear transforms: for Y = AX + b, E(Y) = Aμ + b and Cov(Y) = AΣA′. This one result generates most of multivariate theory.
• Sample analogues: x̄ = (1/n)Σ x_i; S = (1/(n−1)) Σ (x_i − x̄)(x_i − x̄)′ (divisor n−1 for unbiasedness; some texts use n and call it S_n).
• Mahalanobis distance D² = (x − μ)′ Σ⁻¹ (x − μ) measures distance in standard-deviation units allowing for correlation — unlike Euclidean distance it is invariant under non-singular linear transformation.
• Partial correlation r_12.3 measures association between X₁ and X₂ after removing the linear effect of X₃; multiple correlation R_1.23 is the maximum correlation of X₁ with any linear function of X₂, X₃, and lies in [0, 1].

Multivariate normal distribution and sampling distributions

• Density (p-variate, Σ positive definite): depends on x only through the quadratic form (x−μ)′Σ⁻¹(x−μ), so contours of equal density are concentric ellipsoids centred at μ with axes along the eigenvectors of Σ.
• Key properties: (i) all marginals and all conditionals are normal; (ii) every linear combination a′X is univariate normal; (iii) for the normal — and only for the normal — zero covariance implies independence; (iv) X₁ ⫫ X₂ iff Σ₁₂ = 0 for a partition of X.
• Conditional distribution: X₁ | X₂ = x₂ ~ N(μ₁ + Σ₁₂Σ₂₂⁻¹(x₂ − μ₂), Σ₁₁ − Σ₁₂Σ₂₂⁻¹Σ₂₁). The mean is the population regression function; the conditional covariance does not depend on x₂ (homoscedasticity).
• MLEs: μ̂ = x̄ and Σ̂ = (1/n)Σ(x_i − x̄)(x_i − x̄)′; (x̄, S) are jointly sufficient and x̄ ~ N_p(μ, Σ/n), independent of S.
• Wishart distribution: (n−1)S ~ W_p(Σ, n−1) — the multivariate analogue of the chi-square; it is the distribution of Σ Z_i Z_i′ for i.i.d. N_p(0, Σ) vectors.
• Hotelling's T²: T² = n(x̄ − μ₀)′ S⁻¹ (x̄ − μ₀) is the analogue of the one-sample t². Under H₀, [(n − p)/((n − 1)p)] T² ~ F(p, n − p).
• Invariance: T² is invariant under non-singular linear transformations, and is the likelihood-ratio test for the mean vector when Σ is unknown; if Σ is known, use the chi-square statistic n(x̄ − μ₀)′Σ⁻¹(x̄ − μ₀) ~ χ²_p.

Tests on mean vectors and MANOVA

• One sample, Σ known: χ² test as above. Σ unknown: Hotelling's T².
• Two independent samples (equal Σ): pooled S_p = [(n₁−1)S₁ + (n₂−1)S₂]/(n₁+n₂−2); T² = [n₁n₂/(n₁+n₂)](x̄₁ − x̄₂)′ S_p⁻¹ (x̄₁ − x̄₂), with [(n₁+n₂−p−1)/((n₁+n₂−2)p)] T² ~ F(p, n₁+n₂−p−1).
• Paired data: take difference vectors d_i and apply the one-sample T² with μ₀ = 0.
• Confidence region for μ: the ellipsoid n(x̄ − μ)′S⁻¹(x̄ − μ) ≤ [(n−1)p/(n−p)] F_α(p, n−p). Simultaneous intervals come from Roy's union–intersection (T²) projections; Bonferroni intervals are usually shorter for few comparisons.
• MANOVA (one-way, g groups): partition the total SSCP matrix T = B + W (between + within). Test H₀: μ₁ = … = μ_g.
• Test statistics from the eigenvalues λ_i of W⁻¹B: Wilks' Λ = |W|/|W+B| = Π 1/(1+λ_i) (small Λ rejects), Pillai's trace, Hotelling–Lawley trace, Roy's greatest root. Bartlett's chi-square approximation: −[n − 1 − (p+g)/2] ln Λ ~ χ² with p(g−1) df.
• Pillai's trace is the most robust to departures from normality and heterogeneity; Roy's root is most powerful when group means differ along one dimension only.
• Assumptions: independent observations, multivariate normality, equal covariance matrices (checked by Box's M test, itself sensitive to non-normality).
• Other standard tests: equality of covariance matrices (Box's M / Bartlett's), sphericity (Σ = σ²I), and test of independence of sub-vectors (Σ₁₂ = 0) via the likelihood-ratio statistic based on |S|/(|S₁₁||S₂₂|).

Principal component analysis and factor analysis

• PCA idea: find orthogonal linear combinations Y_i = e_i′X of maximum variance. The i-th PC uses the eigenvector e_i of Σ for the i-th largest eigenvalue λ_i; Var(Y_i) = λ_i, Cov(Y_i, Y_j) = 0.
• Variance accounted for by the first k components = (λ₁+…+λ_k)/(λ₁+…+λ_p), since Σλ_i = tr(Σ). Retention rules: cumulative 70–90%, Kaiser's rule (eigenvalue > 1, for correlation-matrix PCA), elbow of the scree plot.
• Loadings: Corr(Y_i, X_k) = e_ik √λ_i / √σ_kk. PCA on Σ is not scale-invariant, so with variables in different units (₹, kg, years) use R — i.e. standardise first.
• PCA is a descriptive rotation, needs no distributional assumption, and is used for dimension reduction, multicollinearity control, index building and image/signal compression.
• Factor analysis model: X − μ = L F + ε, with m common factors F (E F = 0, Cov F = I), loadings L (p × m) and specific errors ε with Cov = Ψ diagonal. Hence Σ = LL′ + Ψ.
• Communality h_i² = Σ_j l_ij² (variance of X_i explained by common factors); uniqueness = ψ_i = σ_ii − h_i².
• Estimation: principal factor (principal axis) method and maximum likelihood (which allows a chi-square test of the number of factors). Loadings are determined only up to an orthogonal rotation — varimax (orthogonal) and oblimin/promax (oblique) rotations aid interpretation.
• Adequacy checks: KMO measure (above ~0.6 acceptable) and Bartlett's test of sphericity (should reject, i.e. R ≠ I).
• PCA vs factor analysis: PCA explains total variance with components as exact functions of the variables; FA is a model explaining only common (shared) variance and treats factors as latent causes.

Classification, association and clustering

• Fisher's linear discriminant (two groups): choose a′x maximising the ratio of between to within separation; a ∝ S_p⁻¹(x̄₁ − x̄₂). Classify to group 1 if a′x exceeds the midpoint a′(x̄₁+x̄₂)/2 (adjusted by log of prior-cost ratio).
• Equivalently, allocate to the group with smaller Mahalanobis distance; LDA assumes equal covariance matrices, QDA allows Σ₁ ≠ Σ₂ and gives a quadratic boundary.
• Performance: apparent error rate is optimistic; use cross-validation / jackknife (leave-one-out) or a hold-out sample. Report the confusion matrix, sensitivity and specificity.
• Canonical correlation: finds pairs (a′X, b′Y) with maximum correlation; the squared canonical correlations are eigenvalues of Σ₁₁⁻¹Σ₁₂Σ₂₂⁻¹Σ₂₁. The number of pairs is min(p, q); the first canonical correlation ≥ |any simple correlation| between the sets.
• Special cases: with q = 1 the first canonical correlation is the multiple correlation R; with p = q = 1 it is |r|.
• Cluster analysis (unsupervised, no group labels): hierarchical — single, complete, average linkage, centroid and Ward's method, shown as a dendrogram; non-hierarchical — k-means, minimising within-cluster sum of squares for a chosen k.
• Choosing k: elbow of the within-SS curve, silhouette width, or dendrogram cut height. Standardise variables first, since distance depends on units.
• Multidimensional scaling (MDS) reconstructs a low-dimensional map from a distance/dissimilarity matrix; correspondence analysis does the analogue for contingency tables.
• Classification vs clustering vs discrimination: clustering discovers groups; discriminant analysis describes known group differences; classification allocates a new unit to a known group.

Practice, software and Indian context

• Assumption order of work: screen for missing values and outliers (large Mahalanobis D²), check normality (Q–Q plot of D², Mardia's skewness and kurtosis tests), check linearity and multicollinearity (VIF > 10, small |R|), then fit.
• Sample size: at least n > p, and in practice n ≥ 5–10 observations per variable for stable S⁻¹; S is singular when n ≤ p, which blocks T², MANOVA and LDA (regularised or PCA-first methods are then used).
• Software: R (stats::prcomp, factanal, MASS::lda, manova, cluster), Python (scikit-learn, statsmodels), SPSS, SAS, Stata, JMP; MS Excel for small PCA via matrix functions. Indian university practicals are mostly in R or SPSS.
• Official and regulatory use: MoSPI/NSO uses multivariate and composite-index methods in NSS and Periodic Labour Force Survey analysis; NITI Aayog's SDG India Index and multidimensional poverty index aggregate many indicators; RBI expects banks and NBFCs to validate credit-scoring and internal-rating models, and the Credit Information Companies (Regulation) Act, 2005 governs bureau scores built on such models. Specific model-validation circular numbers change frequently, so check the current RBI master direction rather than citing an old one.
• Data protection: profiling and segmentation on personal data in India fall under the Digital Personal Data Protection Act, 2023; its rules were notified in 2025 with phased commencement, so confirm the current compliance date before relying on it.
• Standards: ISO 16269 series covers statistical interpretation of data, and BIS publishes Indian Standards on statistical methods; quote the current edition after checking, as several older numbers are withdrawn.

Key points

• Multivariate analysis - simultaneous study of p ≥ 2 correlated variables through μ and Σ.
• Σ properties - symmetric, non-negative definite; tr(Σ) = total variance, |Σ| = generalised variance.
• Linear transform rule - Y = AX + b gives mean Aμ + b and covariance AΣA′.
• Mahalanobis D² - correlation-adjusted, unit-free distance; basis of LDA and multivariate outlier detection.
• MVN contours - ellipsoids with axes along eigenvectors of Σ and half-lengths proportional to √λ_i.
• Zero covariance ⇒ independence - true only under multivariate normality.
• Conditional MVN - mean linear in the conditioning variables, covariance free of them.
• Wishart - distribution of (n−1)S; multivariate analogue of chi-square.
• Hotelling's T² - one- and two-sample test on mean vectors; converts to an F statistic.
• Wilks' Λ - |W|/|W+B| in MANOVA; small values reject equality of mean vectors.
• Box's M - tests equality of covariance matrices; very sensitive to non-normality.
• PCA - orthogonal components, variance λ_i, chosen for dimension reduction; use R when units differ.
• Kaiser and scree rules - retain eigenvalues > 1 or components before the elbow.
• Factor analysis - Σ = LL′ + Ψ; models common variance, rotation-indeterminate, needs interpretation.
• Communality - share of a variable's variance explained by common factors.
• Fisher LDA - direction S_p⁻¹(x̄₁ − x̄₂); equal-Σ assumption, else use QDA.
• Canonical correlation - maximum correlation between linear combinations of two variable sets.
• Clustering - unsupervised grouping; hierarchical (dendrogram) or k-means (fixed k), after standardising.

Formulas and examples

Example 1 — Mahalanobis distance and marginal misjudgement. A bank records X₁ = monthly income (₹ '000) and X₂ = EMI outgo (₹ '000) with μ = (60, 20)′ and Σ = [[100, 40],[40, 25]]. Applicant x = (70, 30)′.

• d = x − μ = (10, 10)′; |Σ| = 100(25) − 40² = 2500 − 1600 = 900.
• Σ⁻¹ = (1/900)[[25, −40],[−40, 100]].
• Σ⁻¹d = (1/900)(25×10 − 40×10, −40×10 + 100×10)′ = (1/900)(−150, 600)′.
• D² = d′Σ⁻¹d = (1/900)(10×(−150) + 10×600) = (1/900)(4500) = 5.0, so D = 2.24.
• Each variable alone is only 1 SD (10/√100) and 2 SD (10/√25) out, but the pair is unusual jointly: for p = 2, P(χ²₂ > 5.0) ≈ 0.082 — borderline, flag for review.

Example 2 — One-sample Hotelling's T². n = 20 trainees tested on p = 2 aptitude scores; x̄ = (52, 48)′, hypothesised μ₀ = (50, 50)′, S = [[16, 6],[6, 9]].

• d = x̄ − μ₀ = (2, −2)′; |S| = 16(9) − 36 = 144 − 36 = 108.
• S⁻¹ = (1/108)[[9, −6],[−6, 16]].
• S⁻¹d = (1/108)(9×2 + (−6)(−2), −6×2 + 16×(−2))′ = (1/108)(30, −44)′.
• d′S⁻¹d = (1/108)(2×30 + (−2)(−44)) = (1/108)(148) = 1.3704.
• T² = n × 1.3704 = 20 × 1.3704 = 27.41.
• F = [(n − p)/((n − 1)p)]T² = (18/38) × 27.41 = 12.98 on (2, 18) df. F₀.₀₅(2,18) ≈ 3.55 → reject H₀; the mean vector differs from (50, 50).

Example 3 — PCA on a correlation matrix. Three standardised test scores with R = [[1, 0.8, 0.8],[0.8, 1, 0.8],[0.8, 0.8, 1]] (equicorrelated, p = 3, ρ = 0.8).

• For an equicorrelated matrix: λ₁ = 1 + (p − 1)ρ = 1 + 2(0.8) = 2.6; λ₂ = λ₃ = 1 − ρ = 0.2 each. Check: sum = 2.6 + 0.2 + 0.2 = 3 = tr(R). ✔
• First PC = (X₁ + X₂ + X₃)/√3, explaining 2.6/3 = 86.7% of total variance — a single "general ability" index suffices.
• Kaiser's rule retains only PC1 (λ > 1). Loading of each variable on PC1 = e_i1√λ₁ = (1/√3)(√2.6) = 0.931.

Example 4 — Fisher's linear discriminant. Two credit classes, p = 2 (income ₹ '000, credit-utilisation %): x̄₁ = (60, 30)′ (good), x̄₂ = (45, 50)′ (default), pooled S_p = [[25, 0],[0, 100]].

• x̄₁ − x̄₂ = (15, −20)′; S_p⁻¹ = [[0.04, 0],[0, 0.01]].
• a = S_p⁻¹(x̄₁ − x̄₂) = (0.04 × 15, 0.01 × (−20))′ = (0.6, −0.2)′.
• Scores: a′x̄₁ = 0.6(60) − 0.2(30) = 36 − 6 = 30; a′x̄₂ = 0.6(45) − 0.2(50) = 27 − 10 = 17.
• Cut-off (equal priors and costs) = (30 + 17)/2 = 23.5. An applicant x = (50, 35)′ scores 0.6(50) − 0.2(35) = 30 − 7 = 23, just below 23.5 → classify as default risk.
• Separation: D² = (x̄₁−x̄₂)′S_p⁻¹(x̄₁−x̄₂) = 15(0.6) + (−20)(−0.2) = 9 + 4 = 13, so D = 3.61 — well-separated groups.

Example 5 — Wilks' Λ in one-way MANOVA. g = 3 teaching methods, p = 2 outcome scores, n = 30 students in all. Eigenvalues of W⁻¹B: λ₁ = 0.50, λ₂ = 0.20.

• Λ = 1/(1+0.50) × 1/(1+0.20) = (0.6667)(0.8333) = 0.5556.
• ln Λ = −0.5878; multiplier = n − 1 − (p + g)/2 = 30 − 1 − 2.5 = 26.5.
• χ² = −26.5 × (−0.5878) = 15.58, df = p(g − 1) = 2 × 2 = 4. χ²₀.₀₅(4) = 9.49 → reject: the three methods differ in mean outcome vector.

Where it is used

• Banking, NBFC and fintech: credit scorecards and internal rating models using logistic/discriminant analysis and PCA on bureau variables (SBI, HDFC Bank, Bajaj Finserv, CIBIL/CRIF analytics teams).
• Market research and FMCG/telecom: customer segmentation by k-means, brand-perception maps from factor analysis and MDS, conjoint and churn models (Nielsen, Kantar, Airtel, Jio, HUL).
• Health, psychometrics and education: validating scales and question papers (factor analysis, Cronbach's alpha), NFHS and ICMR studies, NCERT/NTA item analysis.
• Official statistics and policy: MoSPI/NSO survey analysis, NITI Aayog SDG India Index and multidimensional poverty indices, state-level composite development indices.
• Manufacturing and process industry: multivariate process monitoring with Hotelling T² charts and principal-component-based monitoring in pharma and refining; near-infrared spectral calibration via principal component regression and PLS.
• Job roles: data analyst/scientist, risk analyst, biostatistician, market-research analyst, research officer in ISS and RBI/NABARD, with analytics firms such as Mu Sigma, Fractal, ZS Associates and Quantiphi recruiting on these methods.

Interview questions

• Q: Why not run p separate t-tests instead of one Hotelling's T²? A: Separate tests inflate the overall Type I error and ignore correlations among the variables; T² controls the error rate once and can detect differences that appear only in a linear combination.
• Q: What does Mahalanobis distance add over Euclidean distance? A: It divides by variability and removes correlation through Σ⁻¹, making it unit-free and invariant under linear transformation, so it measures genuine statistical distance.
• Q: State the main difference between PCA and factor analysis. A: PCA is a variance-maximising rotation that explains total variance with components built from the variables; factor analysis is a model (Σ = LL′ + Ψ) that explains only shared variance through latent factors.
• Q: Should PCA be done on the covariance or correlation matrix? A: Use the correlation matrix when variables have different units or very different variances, since PCA on Σ is dominated by the largest-variance variable; use Σ only when units are comparable.
• Q: How many principal components should be kept? A: Enough for 70–90% cumulative variance, or eigenvalues above 1 for correlation-matrix PCA, or the components before the scree-plot elbow — cross-checked against interpretability.
• Q: What is Wilks' lambda and when does it reject? A: Λ = |W|/|W+B| in MANOVA; it is small when between-group dispersion is large relative to within-group, so small values reject equality of mean vectors.
• Q: What assumption does LDA make that QDA does not? A: LDA assumes the groups share one covariance matrix, giving a linear boundary; QDA allows different covariance matrices and gives a quadratic boundary at the cost of many more parameters.
• Q: What is the Wishart distribution used for? A: It is the sampling distribution of the sample SSCP matrix (n−1)S under multivariate normality, and underlies T², Wilks' Λ and tests on covariance matrices.
• Q: Difference between cluster analysis and discriminant analysis? A: Clustering is unsupervised — groups are unknown and must be found; discriminant analysis is supervised — group labels are known and used to build a classification rule.
• Q: What happens if n ≤ p? A: S is singular, so S⁻¹ does not exist and T², MANOVA and LDA fail; the usual fixes are reducing dimension first (PCA) or using regularised/shrinkage estimators.
• Q: Why is factor loading indeterminate? A: For any orthogonal matrix T, (LT)(LT)′ = LL′, so L and LT fit Σ equally well; rotation such as varimax is chosen for interpretability, not for fit.
• Q: What does the first canonical correlation tell you? A: The largest possible correlation between a linear combination of one variable set and a linear combination of the other — it is at least as large as any simple correlation across the sets.

Common mistakes

• Treating zero covariance as independence outside the multivariate normal case — it only guarantees no linear association.
• Running PCA on raw data with mixed units (₹, kg, %) and then reading the first component as meaningful; standardise or use R.
• Confusing PCA with factor analysis, or reporting "communality" for principal components.
• Using χ²_p instead of the F form of T² when Σ is estimated from the sample, and forgetting the multiplier (n − p)/((n − 1)p).
• Arithmetic slips in inverting 2 × 2 matrices — forgetting the 1/|S| factor or the sign change on off-diagonals.
• Reporting the apparent (training) error rate of a discriminant rule as its accuracy instead of a cross-validated rate.
• Ignoring assumption checks — unequal covariance matrices, outliers with large D², or n too small relative to p — and reading a significant MANOVA as proof that every variable differs (follow-up univariate or discriminant analysis is needed).

Related topics

• Statistical quality control (multivariate control charts) · Multiple linear regression and multicollinearity · Sampling distributions and tests of hypotheses · Matrix algebra: eigenvalues and quadratic forms · Machine learning: classification and clustering · Design of experiments and ANOVA