Statistical inference and estimation

In short

Statistical inference is the business of drawing conclusions about an unknown population parameter from a finite sample, and estimation is the half of it that produces a number (a point estimate) or a range (an interval estimate) for that parameter along with a stated uncertainty. It underpins everything a data scientist reports — a click-through rate, a model coefficient, a churn probability, a measured accuracy — because each is computed from a sample and therefore carries sampling error. Indian university syllabi (AICTE model curriculum for AI&DS, UGC CBCS statistics papers) and GATE DA/CS examine estimator properties, maximum likelihood, the Cramér–Rao bound and confidence intervals as standard; placement tests at Mu Sigma, Fractal, ZS Associates, Tiger Analytics, Flipkart, Swiggy, Jio and the analytics arms of TCS/Infosys/Wipro ask for the standard error of a mean or proportion and the meaning of a 95% interval almost routinely. The topic sits upstream of hypothesis testing (which uses these sampling distributions to decide), and here the focus is on *constructing and judging estimators*, not on testing claims.

Explained

Population, sample and the logic of inference

• Population: the entire set of units of interest, described by unknown fixed constants called parameters (μ, σ², p, λ, θ).
• Sample: n units drawn from it; any function of the sample alone is a statistic (x̄, s², p̂). A statistic used to estimate a parameter is an estimator; its computed value on one dataset is an estimate.
• Random sample (iid): X₁…Xₙ independent and identically distributed as the population. Non-random sampling invalidates every formula below.
• Sampling distribution: the probability distribution of a statistic over all possible samples of size n. Inference is nothing but knowing this distribution.
• Standard error (SE): the standard deviation of the sampling distribution. SE of x̄ = σ/√n — the famous √n law: to halve uncertainty you need four times the data.
• Two branches: *estimation* (what is θ?) and *testing* (is θ = θ₀?). Classical/frequentist inference treats θ as fixed and the data as random; Bayesian inference reverses this.
• Sampling designs used in Indian official statistics (NSSO/MoSPI surveys such as PLFS and HCES): simple random sampling with/without replacement, stratified, systematic, cluster and multi-stage; stratification reduces variance when strata are internally homogeneous. Survey weights must be applied before any estimate is quoted.

Properties of a good estimator

Let θ̂ estimate θ.

• Unbiasedness: E[θ̂] = θ for every θ. Bias(θ̂) = E[θ̂] − θ. x̄ is unbiased for μ; the divisor (n−1) in s² makes it unbiased for σ².
• Consistency: θ̂ₙ → θ in probability as n → ∞. Sufficient condition: bias → 0 and Var → 0. An estimator can be biased yet consistent (e.g. MLE of σ² with divisor n).
• Efficiency: among unbiased estimators, the one with smallest variance is most efficient; relative efficiency = Var(θ̂₂)/Var(θ̂₁).
• MVUE / UMVUE: minimum-variance unbiased estimator — unbiased and attains the lowest variance for all θ.
• Sufficiency: T(X) is sufficient for θ if the conditional distribution of the sample given T does not depend on θ — T carries all the information about θ. Checked by the factorisation (Fisher–Neyman) theorem: f(x|θ) = g(T(x), θ)·h(x).
• Completeness + Lehmann–Scheffé: an unbiased function of a complete sufficient statistic is the unique MVUE.
• Rao–Blackwell theorem: conditioning any unbiased estimator on a sufficient statistic never increases variance — E[θ̂|T] is at least as good.
• Mean squared error: MSE = Var + Bias². This is the bias–variance trade-off: a slightly biased, low-variance estimator (ridge regression, shrinkage) can beat the unbiased one in MSE.
• Asymptotic properties: MLEs are consistent, asymptotically unbiased, asymptotically normal and asymptotically efficient under regularity conditions.
• Robustness: the median has breakdown point 50% against outliers, the mean 0% — relevant for Indian income/price data, which are heavily right-skewed.

Methods of point estimation

1. Method of moments (MoM): equate the first k sample moments to the population moments and solve for the k parameters. Simple, needs no likelihood, but can give estimates outside the parameter space and is usually less efficient.
2. Maximum likelihood estimation (MLE):
   • Likelihood L(θ) = ∏ f(xᵢ|θ) treated as a function of θ with data fixed; maximise ℓ(θ) = ln L(θ).
   • Steps: write L → take ln → differentiate (score function) → set dℓ/dθ = 0 → solve → verify d²ℓ/dθ² < 0 → check boundary cases.
   • Invariance: if θ̂ is the MLE of θ, g(θ̂) is the MLE of g(θ).
   • MLEs may be biased (σ̂²_MLE = Σ(xᵢ−x̄)²/n) and may not exist in closed form (logistic regression, Gaussian mixtures → solved numerically by Newton–Raphson, IRLS or EM).
   • Minimising cross-entropy loss in a neural classifier *is* maximum likelihood for a categorical model; least squares is MLE under Gaussian errors.
3. Least squares: minimise Σ(yᵢ − ŷᵢ)²; by the Gauss–Markov theorem OLS is BLUE (best linear unbiased) without any normality assumption.
4. Bayesian point estimates: posterior mean/median/mode (MAP); MAP = MLE plus a penalty from the prior (Gaussian prior ⇒ L2/ridge, Laplace prior ⇒ L1/lasso). Detail belongs to the Bayesian topic.
5. Method of maximum entropy, minimum chi-square, M-estimation (Huber): alternatives used when robustness or constraints matter.

Fisher information and the Cramér–Rao lower bound

• Score: U(θ) = ∂ℓ/∂θ, with E[U] = 0.
• Fisher information: I(θ) = E[U²] = −E[∂²ℓ/∂θ²] for one observation; for n iid observations Iₙ(θ) = n·I(θ). It measures the curvature of the log-likelihood — sharp peak ⇒ precise estimate.
• Cramér–Rao lower bound (CRLB): for any unbiased θ̂, Var(θ̂) ≥ 1/Iₙ(θ). An estimator meeting it is efficient; it exists only for exponential-family models.
• Regularity conditions: support independent of θ, differentiability, interchange of derivative and integral. The bound fails for U(0, θ) (support depends on θ), a favourite exam trap.
• Observed information at θ̂ gives the usual asymptotic SE: SE(θ̂) ≈ 1/√(−ℓ″(θ̂)), the basis of the standard errors printed by statsmodels, R and scikit-learn wrappers.

Interval estimation (confidence intervals)

• Definition: a random interval (L, U) with P(L ≤ θ ≤ U) = 1 − α before the data are seen. 1 − α is the confidence level (typically 95% ⇒ α = 0.05).
• Correct reading: 95% of such intervals, over repeated sampling, contain the fixed θ. It is *not* "a 95% probability that θ lies in this computed interval" — that statement is Bayesian (a credible interval).
• Pivotal method: find a quantity whose distribution is free of θ (Z, t, χ², F) and invert the probability statement.
• Which sampling distribution:
  • σ known, or n large (CLT): Z interval.
  • σ unknown, normal population, small n: t interval with ν = n − 1 degrees of freedom (Student's t — heavier tails, wider interval).
  • Variance of a normal population: χ² with n − 1 df (asymmetric interval).
  • Ratio of two variances: F distribution.
  • Proportion: Wald interval p̂ ± z√(p̂q̂/n) for large n; Wilson or Agresti–Coull when np̂ < 10 or p̂ near 0 or 1; Clopper–Pearson exact.
• Central limit theorem: x̄ is approximately normal for large n whatever the population shape (rule of thumb n ≥ 30, more for very skewed data); this is why Z/t intervals work so widely.
• Finite population correction √((N−n)/(N−1)) applies when n/N > 0.05, as in audit sampling.
• Width = 2 × (critical value) × SE; it shrinks with √n, grows with σ and with confidence level. Margin of error (MoE) = half-width — the ±2% quoted with Indian opinion polls.
• Tolerance interval covers a proportion of *individual* values; a prediction interval covers a future observation and is wider than the confidence interval for the mean.
• Bootstrap: resample the data with replacement B times (B = 1,000–10,000), recompute the statistic, take percentiles 2.5 and 97.5. Distribution-free; essential for medians, AUC, Gini and model metrics where no formula exists.
• Delta method: Var(g(θ̂)) ≈ [g′(θ)]² Var(θ̂) — gives SEs for ratios, odds and log-transformed quantities.

Sample size, reporting and Indian practice

• Choose n so that the margin of error meets a target: n = (z σ / E)² for a mean, n = z² p(1−p)/E² for a proportion (use p = 0.5 for the worst case when p is unknown).
• Report an estimate as estimate ± MoE (confidence level, n), with the sampling design named. MoSPI/NSSO reports publish relative standard errors by stratum; TRAI and RBI statistical releases likewise flag survey-based figures.
• Government statistics in India are collected under the Collection of Statistics Act, 2008 (with the 2017 amendment); official releases follow MoSPI's National Quality Assurance Framework. Clinical and bioequivalence studies submitted to CDSCO must pre-specify sample size and the estimation method (New Drugs and Clinical Trials Rules, 2019).
• Personal-data-driven estimation in India must respect the Digital Personal Data Protection Act, 2023 — aggregate estimates are fine, but purpose limitation and consent apply to the underlying records (rules operationalising the Act were still being phased in at the time of writing, so check the current notification).
• Estimation vs testing: modern practice (and most Indian analytics employers) prefers reporting an effect size with its confidence interval over a bare p-value, because the interval shows both significance and practical magnitude.

Key points

• Parameter vs statistic — a parameter is a fixed unknown constant of the population; a statistic is computed from the sample and is random.
• Estimator vs estimate — the rule/formula vs the one number it produces on a given dataset.
• Standard error — SD of the sampling distribution; SE(x̄) = σ/√n, SE(p̂) = √(pq/n).
• Unbiased — E[θ̂] = θ; s² uses n−1 (Bessel's correction) to be unbiased for σ².
• Consistent — converges in probability to θ as n grows; weaker and more practically important than unbiasedness.
• Efficient — smallest variance among unbiased estimators; attains the CRLB.
• Sufficient statistic — summarises the sample without losing information about θ; found by the factorisation theorem.
• MSE = Variance + Bias² — the single criterion that makes shrinkage and regularisation defensible.
• MLE — maximises the likelihood; consistent, asymptotically normal and efficient, but can be biased in small samples.
• Method of moments — equate sample and population moments; easy, often less efficient.
• Fisher information — curvature of the log-likelihood; Iₙ = n·I₁, and Var ≥ 1/Iₙ.
• Invariance of MLE — the MLE of g(θ) is g of the MLE.
• Confidence interval — a random interval covering θ in 1 − α of repeated samples; not a probability statement about one computed interval.
• t vs Z — t when σ is estimated from a small normal sample (df = n−1); Z when σ is known or n is large.
• χ² interval for a normal variance, F for a ratio of variances — both asymmetric.
• CLT — x̄ tends to normality regardless of population shape, making Z/t intervals broadly valid.
• Bootstrap — resampling gives intervals for statistics with no closed-form SE (median, AUC, Gini).
• Sample size — n ∝ 1/E²; halving the margin of error needs four times the sample.

Formulas and examples

Example 1 — CI for a mean, σ unknown (small sample). A Bengaluru logistics firm times 16 last-mile deliveries: x̄ = 42 min, s = 8 min. Build a 95% CI for the mean delivery time.

• σ unknown, n = 16 small, times roughly normal ⇒ use t with ν = 15; t₀.₀₂₅,₁₅ = 2.131.
• SE = s/√n = 8/4 = 2.000 min.
• Margin = 2.131 × 2.000 = 4.262 min.
• CI = 42 ± 4.26 → (37.74, 46.26) minutes. With z = 1.96 the interval would be (38.08, 45.92) — too narrow, which is exactly the error of ignoring the t correction.

Example 2 — CI and sample size for a proportion. In a survey of n = 1,200 app users, 384 clicked a new banner.

• p̂ = 384/1200 = 0.32; q̂ = 0.68.
• SE = √(0.32 × 0.68 / 1200) = √(0.2176/1200) = √0.00018133 = 0.013466.
• 95% CI = 0.32 ± 1.96 × 0.013466 = 0.32 ± 0.026394 → (0.2936, 0.3464), i.e. 29.4% to 34.6%; MoE ≈ ±2.6 points. Check np̂ = 384 ≫ 10, so Wald is fine.
• To cut the MoE to ±1 point (E = 0.01) at the same p̂: n = 1.96² × 0.32 × 0.68 / 0.01² = 3.8416 × 0.2176 / 0.0001 = 0.83593/0.0001 ≈ 8,360 users — about 7× the sample for a 2.6× tighter interval.

Example 3 — MLE for a Bernoulli/binomial parameter, with its CRLB. n independent users, each converts with probability p; x conversions observed.

• ℓ(p) = x ln p + (n − x) ln(1 − p).
• dℓ/dp = x/p − (n − x)/(1 − p) = 0 ⇒ x(1 − p) = p(n − x) ⇒ x = np ⇒ p̂ = x/n.
• d²ℓ/dp² = −x/p² − (n − x)/(1 − p)² < 0 ⇒ maximum.
• I₁(p) = 1/[p(1 − p)] ⇒ Iₙ = n/[p(1−p)] ⇒ CRLB = p(1 − p)/n. Since Var(x/n) = p(1 − p)/n exactly, p̂ is efficient (an MVUE).
• Numbers: x = 45, n = 300 ⇒ p̂ = 0.15; SE = √(0.15 × 0.85/300) = √0.000425 = 0.020616; 95% CI = 0.15 ± 0.0404 → (0.1096, 0.1904).

Example 4 — MLE vs method of moments for an exponential, and bias of σ̂². Service times (hours) between machine failures: 2, 3, 5, 6, 9 (n = 5), modelled Exponential(λ) with mean 1/λ.

• Σxᵢ = 25, x̄ = 5.
• MoM: population mean 1/λ = x̄ ⇒ λ̂ = 1/5 = 0.20 per hour.
• MLE: ℓ(λ) = n ln λ − λΣxᵢ; dℓ/dλ = n/λ − Σxᵢ = 0 ⇒ λ̂ = n/Σxᵢ = 5/25 = 0.20 per hour — the two coincide here (they do not in general).
• Mean life by invariance: 1/λ̂ = 5 hours.
• Now the variance of the same data: deviations −3, −2, 0, 1, 4 ⇒ Σ(xᵢ−x̄)² = 9 + 4 + 0 + 1 + 16 = 30. s² = 30/4 = 7.5 (unbiased); σ̂²_MLE = 30/5 = 6.0 — biased low by factor (n−1)/n = 0.8, a 20% understatement at n = 5 and only 1% at n = 100.

Example 5 — CI for a normal variance (χ²). A Coimbatore unit measures the diameter of 25 shafts: s = 0.040 mm. 95% CI for σ², ν = 24, χ²₀.₀₂₅,₂₄ = 39.364, χ²₀.₉₇₅,₂₄ = 12.401.

• (n−1)s² = 24 × 0.0016 = 0.0384 mm².
• Lower = 0.0384/39.364 = 0.0009755 mm²; Upper = 0.0384/12.401 = 0.0030965 mm².
• CI for σ² = (0.000976, 0.003097) mm² ⇒ for σ: (0.0312, 0.0557) mm. Note the asymmetry about s = 0.040 — a classic exam point.

import numpy as np
from scipy import stats

x = np.array([2, 3, 5, 6, 9], dtype=float)
n = x.size
mean, s = x.mean(), x.std(ddof=1)          # ddof=1 -> unbiased s^2
se = s / np.sqrt(n)
t_crit = stats.t.ppf(0.975, df=n - 1)
print(mean, s**2, (mean - t_crit*se, mean + t_crit*se))

# bootstrap CI for the median - no closed-form SE
rng = np.random.default_rng(0)
meds = [np.median(rng.choice(x, n, replace=True)) for _ in range(10000)]
print(np.percentile(meds, [2.5, 97.5]))

Where it is used

• E-commerce and consumer internet (Flipkart, Myntra, Swiggy, Zomato, Meesho): estimating conversion rates, delivery-time means and revenue per user with confidence intervals before a feature is rolled out.
• Banking, insurance and credit (HDFC, ICICI, SBI, Bajaj Finance, LIC, Policybazaar): estimating default probabilities (PD) by MLE in logistic scorecards, loss severities, and reserve estimates; RBI model-validation documentation asks for parameter SEs and stability.
• Pharma and clinical research (Sun Pharma, Dr Reddy's, Biocon, CRO/CDSCO submissions): sample-size determination, confidence intervals for treatment effects and bioequivalence ratios.
• Manufacturing quality and metrology (Tata Motors, Bosch, L&T, NABL-accredited labs): estimating process mean and σ from samples for control limits and capability indices; measurement uncertainty is reported as an expanded uncertainty (coverage factor k ≈ 2 for ~95%).
• Official statistics and policy (MoSPI/NSSO PLFS, HCES, NFHS, CMIE, RBI consumer confidence surveys): stratified multi-stage estimation with weights and published relative standard errors.
• ML and MLOps roles: reporting model accuracy or AUC with a bootstrap interval, estimating coefficients and their SEs, calibrating predicted probabilities, and sizing evaluation sets — asked of data scientists, quantitative analysts, risk modellers and ML engineers.

Interview questions

• Q: What is the difference between an estimator and an estimate? A: An estimator is the rule or formula (a random variable, e.g. x̄); an estimate is the specific value it takes on an observed sample (e.g. 42 minutes).
• Q: Why does sample variance divide by n − 1? A: Deviations are taken about x̄, not μ, which costs one degree of freedom; dividing by n − 1 makes E[s²] = σ² exactly.
• Q: What does a 95% confidence interval actually mean? A: If the sampling and interval procedure were repeated many times, about 95% of the intervals produced would contain the fixed true parameter; it is not a 95% probability for the one interval in hand.
• Q: Can an estimator be biased but still preferable? A: Yes — MSE = variance + bias², so a low-variance biased estimator such as a ridge or shrinkage estimator can have smaller MSE than the unbiased one.
• Q: Unbiasedness or consistency — which matters more in practice? A: Consistency, because with enough data a consistent estimator converges to the truth, while an unbiased estimator with huge variance can be useless at any single sample size.
• Q: State the Cramér–Rao lower bound and when it fails. A: For any unbiased θ̂, Var(θ̂) ≥ 1/Iₙ(θ); it fails when the regularity conditions break, for example U(0, θ) where the support depends on θ.
• Q: Derive the MLE of p for a Bernoulli sample. A: ℓ = x ln p + (n−x) ln(1−p); setting dℓ/dp = 0 gives x/p = (n−x)/(1−p) ⇒ p̂ = x/n, and the second derivative is negative so it is a maximum.
• Q: When would you use t instead of Z? A: When σ is unknown and estimated by s from a small (roughly n < 30) sample from an approximately normal population; df = n − 1.
• Q: MLE vs method of moments? A: MoM equates sample and population moments — quick but often less efficient; MLE maximises the likelihood and is asymptotically efficient, but needs the full distribution and may require numerical optimisation.
• Q: What is a sufficient statistic, and why care? A: A statistic whose conditional sample distribution is free of θ, so it captures all information about θ; by Rao–Blackwell and Lehmann–Scheffé it leads to the MVUE and allows massive data reduction (Σxᵢ instead of all xᵢ).
• Q: How do you get a confidence interval for the median or AUC? A: Bootstrap — resample with replacement a few thousand times, recompute the statistic and take the 2.5th and 97.5th percentiles.
• Q: How many users do you need to estimate a conversion rate within ±1% at 95% confidence? A: n = z²p(1−p)/E²; with the worst case p = 0.5, n = 3.8416 × 0.25/0.0001 ≈ 9,604 users.
• Q: Why does a prediction interval exceed a confidence interval for the mean? A: The confidence interval covers only the uncertainty in the mean (σ/√n), while a prediction interval must also absorb the variability σ of a single future observation.

Common mistakes

• Saying "there is a 95% probability the parameter lies in this interval" — that is a Bayesian credible interval, not a frequentist confidence interval.
• Using Z instead of t for a small sample with σ estimated, producing intervals that are too narrow and over-confident.
• Dividing by n instead of n − 1 for s², or mixing the two (np.std defaults to ddof = 0, so pass ddof = 1).
• Confusing standard deviation (spread of data) with standard error (spread of the statistic), and quoting one where the other belongs.
• Building a symmetric interval for a variance or for a proportion near 0 or 1 — use χ² for σ² and Wilson/Clopper–Pearson for extreme p̂.
• Treating a larger sample as a cure for bias: non-random or self-selected samples converge to the wrong value, and no √n gain fixes that.
• Forgetting the invariance rule and re-deriving the MLE of a function of θ, or ignoring boundary solutions where the derivative is never zero (e.g. the MLE of θ in U(0, θ) is max xᵢ).

Related topics

Hypothesis testing · Probability distributions · Descriptive statistics · Bayesian statistics · Experiment design and A/B testing · Causal inference basics