Dimensionality reduction and PCA

In short

Dimensionality reduction maps data with p features into k ≪ p new features that keep most of the information, either by selecting features or by building new ones as combinations of the old. Principal Component Analysis (PCA) is the standard linear method: it finds orthogonal directions of maximum variance, which turn out to be the eigenvectors of the covariance matrix (equivalently the right singular vectors of the centred data matrix). It is examined heavily in university ML and pattern-recognition papers, in GATE CS/DA (linear algebra plus ML sections) and in almost every data-science interview, because eigen-decomposition and variance accounting can be tested with small numbers. In Indian practice it appears in credit-risk scoring, biometric and face recognition, hyperspectral remote sensing at NRSC/ISRO, sensor and telecom monitoring, and as a preprocessing step before regression, clustering or anomaly detection.

Explained

Why reduce dimensions — the curse of dimensionality

• Sample complexity: to keep the same density of points, the number of samples needed grows exponentially with p; a 10-feature grid with 10 bins per feature has 10¹⁰ cells.
• Distance concentration: in high dimensions the ratio (max distance − min distance)/min distance → 0, so nearest-neighbour and k-means distances lose contrast, hurting kNN, SVM with RBF kernels and clustering.
• Multicollinearity: correlated columns make least-squares coefficients unstable (huge variance) and X′X near-singular.
• Cost and noise: fewer dimensions mean faster training, smaller models, less storage, and often less variance because noisy directions are discarded.
• Visualisation: human inspection needs 2–3 dimensions.
• Two families: *feature selection* keeps a subset of original columns (interpretable); *feature extraction / projection* builds new columns (PCA, LDA, autoencoders) which are usually not interpretable one-to-one.

PCA: idea, derivation and algorithm

• Goal: find a unit vector w₁ such that the projected data z = Xw₁ has maximum variance; then w₂ maximising variance subject to w₂ ⊥ w₁, and so on.
• Derivation outline (standard exam answer): variance of projection = wᵀΣw where Σ is the covariance matrix. Maximise wᵀΣw subject to wᵀw = 1. Lagrangian L = wᵀΣw − λ(wᵀw − 1); ∂L/∂w = 2Σw − 2λw = 0 ⇒ Σw = λw. So w is an eigenvector and the variance captured equals its eigenvalue λ. The largest eigenvalue gives PC1.
• Equivalent view (Pearson, 1901): the same k-dimensional subspace minimises the sum of squared perpendicular distances from the points — PCA minimises reconstruction error. The two formulations (max variance, min reconstruction error) give identical answers.
• Algorithm (covariance route):
  1. Centre each column: subtract the mean (mandatory). Standardise (divide by sd) if units differ.
  2. Compute Σ = XᵀX/(n − 1), a p × p symmetric positive semi-definite matrix.
  3. Eigen-decompose Σ: eigenvalues λ₁ ≥ λ₂ ≥ … ≥ λp ≥ 0, orthonormal eigenvectors (the components or loadings).
  4. Pick k components, form W (p × k), and project: Z = XW (the scores).
  5. Reconstruct if needed: X̂ = ZWᵀ + mean.
• Properties: components are orthogonal and the scores are uncorrelated; Σλᵢ = total variance = trace(Σ); PCA is a rotation (orthogonal linear map), so it is reversible when k = p; it is unsupervised — the label y is never used.
• Sign and scale ambiguity: an eigenvector and its negative are both valid, so PC signs may flip between libraries; magnitudes are fixed only by the unit-norm constraint.

SVD route and numerical practice

• For the centred matrix X (n × p), X = UDVᵀ. The columns of V are the PCA components, and λᵢ = σᵢ²/(n − 1) where σᵢ are the singular values; scores Z = UD.
• Libraries (scikit-learn PCA) use SVD, not explicit covariance: better conditioned, avoids forming XᵀX, and works when p > n.
• Eckart–Young theorem: the rank-k truncated SVD is the best rank-k approximation of X in Frobenius and spectral norm — the theoretical guarantee behind PCA compression.
• Variants for scale: *randomised SVD* for large p, *IncrementalPCA* for data larger than RAM, *TruncatedSVD* (also called LSA) which skips centring and so works on sparse term-document matrices.
• Whitening: divide each score by √λᵢ so all components have unit variance — used before ICA and in some neural pipelines; it amplifies low-variance noise directions.

Choosing k, and reading the output

• Explained variance ratio: λᵢ/Σλⱼ; the cumulative curve is the usual criterion — keep enough components for 90%, 95% or 99% depending on the use.
• Scree plot: λ against component number; keep components before the "elbow" where the curve flattens.
• Kaiser criterion: on a *correlation* matrix, retain components with λ > 1 (a component should explain more than one original standardised variable). Rough and known to over-retain.
• Cross-validation / downstream metric: treat k as a hyperparameter and tune it by the validation score of the final model — the most defensible choice in placement answers.
• Loadings interpretation: large absolute loading means that original feature contributes strongly to the component; a biplot shows scores and loading vectors together. Correlation of feature j with PCᵢ = wᵢⱼ√λᵢ/sdⱼ.
• Residual variance of the dropped components equals the mean squared reconstruction error (up to the n/(n−1) factor).

Assumptions, advantages, limitations

• Advantages: no hyperparameters beyond k, deterministic, fast (O(min(np², n²p))), removes correlation, reduces storage, acts as a regulariser and a denoiser, has a closed-form inverse transform (exact only when k = p, otherwise a lossy reconstruction).
• Assumptions / limits:
  • Captures linear structure only; a spiral or a torus is not unfolded by PCA.
  • Assumes variance = information; a low-variance feature may be the one that separates classes (PCA can delete the discriminative direction because it is unsupervised).
  • Scale-sensitive: without standardisation, a feature in rupees swamps a feature in years.
  • Sensitive to outliers (variance is squared-error based) — robust PCA or trimming helps.
  • Not interpretable for compliance or regulatory explanation; a component is a mixture of all features.
  • Mean-centring is essential; skipping it makes PC1 point at the data's mean vector.
  • Does not anonymise data — scores are reversible, so under the Digital Personal Data Protection Act, 2023 PCA output on personal data is still personal data.
• Fit on train only: compute mean, sd and components on the training set and apply the same transform to validation/test; fitting on the full data leaks information.

Other dimensionality-reduction methods (comparisons examiners ask for)

• LDA (Linear Discriminant Analysis / Fisher): supervised; maximises between-class to within-class scatter, J(w) = wᵀS_B w / wᵀS_W w, solved by the eigenvectors of S_W⁻¹S_B. Gives at most C − 1 components for C classes. Better than PCA for classification when classes differ in mean.
• Factor analysis: a generative model x = Λf + μ + ε with per-feature noise ε; explains *covariance*, whereas PCA explains *total variance*. Probabilistic PCA is the special case of equal noise variance.
• ICA: finds statistically independent, not merely uncorrelated, components; used for blind source separation (EEG/ECG artefact removal, audio).
• Kernel PCA: PCA in a feature space via the kernel trick — eigen-decompose the centred kernel matrix K (n × n) instead of Σ; captures non-linear structure but has no simple inverse transform and costs O(n³).
• Manifold learning: MDS (preserves pairwise distances), Isomap (geodesic distances on a kNN graph), LLE, Laplacian eigenmaps.
• t-SNE and UMAP: non-linear, neighbour-preserving, for visualisation — stochastic, no reliable inverse transform, distances between far-apart clusters are not meaningful, and t-SNE's perplexity/UMAP's n_neighbors change the picture. Standard practice: PCA to ~50 dims first, then t-SNE/UMAP to 2.
• Random projection: project onto a random matrix; the Johnson–Lindenstrauss lemma bounds distortion with k ≈ O(log n / ε²) regardless of p — very cheap, data-independent.
• Autoencoders: neural encoder–decoder; a single linear hidden layer with squared loss spans the same subspace as PCA, while deep non-linear autoencoders generalise it.
• Feature hashing / embeddings: used for very high-cardinality categorical and text data where PCA is impractical.

Key points

• Dimensionality reduction — mapping p features to k ≪ p while preserving as much useful structure as possible.
• Curse of dimensionality — exponential sample need and loss of distance contrast as p grows.
• PCA — orthogonal linear projection onto directions of maximum variance; unsupervised.
• Eigen equation — Σw = λw; eigenvector = component direction, eigenvalue = variance along it.
• Total variance — Σλᵢ = trace(Σ) = sum of feature variances; it is redistributed, not created.
• Explained variance ratio — λᵢ/Σλⱼ; cumulative ratio decides k.
• Scores vs loadings — scores Z = XW are the new coordinates; loadings W say how features combine.
• Centring is compulsory; standardising is conditional on features having different units or very different variances.
• SVD — X = UDVᵀ; V = components, λᵢ = σᵢ²/(n − 1); the numerically preferred route.
• Dual objectives — maximum projected variance ≡ minimum squared reconstruction error.
• Reconstruction error — equals the sum of discarded eigenvalues.
• Scree plot / Kaiser rule (λ > 1 on correlation matrix) — quick ways to pick k; cross-validation is the strongest.
• PCA ≠ feature selection — it creates new mixed features, so original-feature interpretability is lost.
• LDA — supervised alternative, at most C − 1 components, maximises class separation.
• Kernel PCA / Isomap / autoencoders — non-linear extensions when the manifold is curved.
• t-SNE and UMAP — visualisation only; stochastic, no honest inverse, global distances unreliable.
• Outlier sensitivity — one extreme point can rotate PC1; inspect and treat outliers first.
• Pipeline discipline — fit scaler and PCA on training data only, then transform the test set.

Formulas and examples

Example 1 — full 2-D PCA by hand (the standard exam question). Marks of 5 students in Maths (M) and Physics (P): M = 4, 8, 13, 7, 18 ; P = 5, 9, 11, 9, 16. Both means are 10.

• Var(M) = 122/4 = 30.5; Var(P) = 64/4 = 16; Cov = 86/4 = 21.5. So Σ = [[30.5, 21.5], [21.5, 16]].
• trace = 46.5, det = 30.5×16 − 21.5² = 488 − 462.25 = 25.75.
• λ = [46.5 ± √(46.5² − 4×25.75)]/2 = [46.5 ± √(2162.25 − 103)]/2 = [46.5 ± √2059.25]/2 = [46.5 ± 45.379]/2.
• λ₁ = 45.94, λ₂ = 0.56 (check: 45.94 + 0.56 = 46.50 = trace ✓).
• Explained variance: PC1 = 45.94/46.50 = 98.8%, PC2 = 1.2%.
• Eigenvector for λ₁: (30.5 − 45.94)w₁ + 21.5w₂ = 0 ⇒ w₂ = 0.7181w₁ ⇒ w = (1, 0.7181)/1.2312 = (0.812, 0.583).
• PC1 scores z = 0.812(M dev) + 0.583(P dev): −7.79, −2.21, 3.02, −3.02, 10.00.
• Check: mean of scores = 0 ✓; variance = (60.68 + 4.88 + 9.12 + 9.12 + 99.96)/4 = 183.76/4 = 45.94 = λ₁ ✓.
• Conclusion: one component replaces two features with 1.2% information loss — PC1 is essentially "overall academic strength".

Example 2 — choosing k from eigenvalues. Correlation-matrix PCA on p = 7 features gives λ = 3.0, 1.6, 1.1, 0.7, 0.3, 0.2, 0.1 (sum = 7 = p, as it must be).

Kaiser rule (λ > 1) keeps 3 components (81.4%); a 90% variance target needs 4 (91.4%); the scree elbow is after PC3. Answer: k = 3 for interpretation, k = 4 if 90% is specified.

Example 3 — why standardisation matters. Two features: annual salary (sd ₹2,00,000 ⇒ variance 4 × 10¹⁰ rupee²) and experience (sd 3 years ⇒ variance 9 year²), correlation 0.4 ⇒ covariance = 0.4 × 2,00,000 × 3 = 2.4 × 10⁵.

• Covariance PCA (unscaled): trace = 4 × 10¹⁰ + 9; λ₁ ≈ 4 × 10¹⁰, so PC1 explains ≈ 99.99999998% and is simply the salary axis — the experience feature is effectively deleted.
• Correlation PCA (standardised): R = [[1, 0.4], [0.4, 1]] ⇒ λ = 1 ± 0.4 = 1.4 and 0.6, i.e. 70% and 30%, with components (0.707, 0.707) and (0.707, −0.707) — a sum and a difference, which is the meaningful result.

Example 4 — compression and reconstruction. 1,000 grayscale face images of 64 × 64 = 4,096 pixels, keeping 50 components ("eigenfaces").

• Raw storage: 1,000 × 4,096 = 4,096,000 values.
• PCA storage: scores 1,000 × 50 = 50,000 + components 50 × 4,096 = 204,800 + mean 4,096 = 258,896 values.
• Compression ≈ 4,096,000/258,896 ≈ 15.8×. If the 50 components carry 92% of the variance, mean squared reconstruction error = 8% of the total pixel variance.

Example 5 — eigenvalues from singular values. Centred X with n = 101 samples has singular values 12, 5, 2.

• λᵢ = σᵢ²/(n − 1) = 144/100 = 1.44, 25/100 = 0.25, 4/100 = 0.04; total = 1.73.
• PC1 share = 1.44/1.73 = 83.2%; PC1+PC2 = 1.69/1.73 = 97.7%, so k = 2 suffices for a 95% target.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95, svd_solver="full"))
Z = pipe.fit_transform(X_train)          # fit on train only
Z_test = pipe.transform(X_test)          # same mean, sd and components
pca = pipe.named_steps["pca"]
print(pca.n_components_, pca.explained_variance_ratio_.cumsum())
X_back = pipe.inverse_transform(Z)       # approximate reconstruction

Where it is used

• Banking, NBFCs and fintech (HDFC, ICICI, Bajaj Finserv, CRED, Paytm): collapsing hundreds of correlated bureau and Account Aggregator variables into a few components before scoring; also PCA-based monitoring of transaction features for fraud. Regulator-facing scorecards usually keep interpretable features, so PCA is used for exploration and noise reduction rather than the final model.
• Remote sensing and space (ISRO/NRSC, Bhuvan, AVIRIS-NG and hyperspectral missions): hundreds of spectral bands are reduced by PCA/MNF before crop, mineral or land-use classification; PC1–PC3 false-colour composites are a standard product.
• Biometrics and vision (UIDAI ecosystem vendors, surveillance and attendance products): eigenfaces and PCA/LDA pipelines for face and fingerprint templates; PCA still used to shrink deep-network embeddings before indexing.
• Manufacturing and process industry (Tata Steel, Reliance, L&T, Honeywell, Siemens India): multivariate statistical process control — PCA on sensor arrays with Hotelling's T² and SPE (Q) charts for drift and fault detection.
• Healthcare and genomics (CSIR-IGIB, AIIMS research groups, diagnostics startups): PCA on gene-expression and population-genetics data (the first two PCs reveal ancestry and batch effects); also on ECG/EEG signals, often with ICA for artefact removal.
• Quant finance and risk (NSE/BSE-linked desks, mutual funds): PCA on yield curves gives the classic level–slope–curvature factors; PCA on equity returns builds risk factor models and reduces covariance-estimation noise.
• Job roles: data scientist, ML engineer, quantitative analyst, remote-sensing/GIS analyst, computer-vision engineer, process data analyst.

Interview questions

• Q: What exactly does PCA maximise? A: The variance of the data projected on each successive orthogonal unit direction; equivalently it minimises the sum of squared perpendicular reconstruction distances onto the chosen subspace.
• Q: Why do eigenvectors of the covariance matrix solve it? A: Maximising wᵀΣw under wᵀw = 1 with a Lagrange multiplier gives Σw = λw, so the stationary directions are eigenvectors and the captured variance is the eigenvalue.
• Q: Must you standardise before PCA? A: Centre always; standardise when features are in different units or have very different variances, otherwise the largest-variance unit dominates PC1 — e.g. salary in rupees against experience in years.
• Q: PCA or SVD in practice? A: They give the same components; SVD on the centred matrix is numerically more stable, avoids forming XᵀX, and works when p > n, which is why libraries use it.
• Q: How do you choose the number of components? A: Cumulative explained variance (90/95/99%), the scree elbow, λ > 1 on a correlation matrix, or by tuning k against the downstream validation metric — the last is the most defensible.
• Q: Can PCA hurt classification accuracy? A: Yes — it is unsupervised, so a low-variance direction that happens to separate the classes can be discarded; LDA or supervised feature selection is safer when that risk exists.
• Q: Difference between PCA and LDA? A: PCA is unsupervised and maximises total variance; LDA uses labels, maximises between-class over within-class scatter and yields at most C − 1 components.
• Q: Are principal components independent? A: They are uncorrelated and orthogonal, which equals independence only for Gaussian data; ICA is the method that targets statistical independence.
• Q: What happens if you forget to mean-centre? A: The first component is pulled towards the data's mean vector rather than the direction of maximum variance, and the "variances" become second moments about the origin.
• Q: How does PCA relate to multicollinearity and regression? A: Components are uncorrelated, so principal component regression removes collinearity and stabilises coefficients, at the cost of interpretability; ridge regression does something similar by shrinking low-eigenvalue directions.
• Q: Does PCA anonymise personal data? A: No — the transform is linear and invertible given the components, so scores remain personal data under the DPDP Act, 2023; use proper anonymisation or differential privacy if that is the requirement.
• Q: Why run PCA before t-SNE? A: PCA to about 50 dimensions removes noise and makes t-SNE far faster and more stable; t-SNE alone on thousands of raw dimensions is slow and more sensitive to its parameters.
• Q: Where does kernel PCA help and what does it cost? A: When the structure is non-linear — it eigen-decomposes the centred n × n kernel matrix; cost is about O(n³) in time and O(n²) in memory, and there is no exact inverse transform.

Common mistakes

• Fitting the scaler or PCA on the full dataset (or on the test set) instead of only the training fold — a classic leakage error flagged in interviews.
• Treating PCA as feature selection and then claiming "feature X was important"; a component mixes all features, so original-feature importance is not recovered that way.
• Applying PCA to one-hot or sparse count data without thought; use TruncatedSVD/LSA for sparse text, and remember centring destroys sparsity.
• Forgetting that eigenvalues must sum to the trace — a quick check that catches arithmetic slips in exam problems; and forgetting to normalise the eigenvector to unit length before computing scores.
• Feeding t-SNE/UMAP coordinates into a classifier as features, or interpreting distances between t-SNE clusters as meaningful.
• Assuming high variance means high predictive value, or expecting PCA to remove non-linear redundancy it cannot see.
• Ignoring outliers before PCA, then wondering why PC1 points at a single extreme record.
• Keeping too few components because the scree elbow looked convincing, without checking the effect on the final model's metric.

Related topics

Feature selection and model interpretability · Clustering: k-means and hierarchical · Overfitting, bias-variance and regularisation · Anomaly detection · Machine learning workflow · Linear algebra: eigenvalues and SVD