Estimation theory

Statistics (B.Sc) · Inference · free study notes for exams, placements and interviews

In short

Estimation theory is the branch of statistical inference that uses a sample to guess the unknown parameters of a population - a single best value (point estimate) or a range with an attached confidence level (interval estimate). It supplies the rules for judging whether a guess is any good: unbiasedness, consistency, efficiency and sufficiency, and the Cramér-Rao bound that fixes how precise any unbiased guess can be. B.Sc Statistics papers on Statistical Inference, the ISS exam, IIT-JAM, CMI/ISI entrance and GATE Statistics (ST) all test it heavily, usually as "derive the MLE / show the estimator is unbiased / build a 95% CI". In work it is the arithmetic behind NSSO and PLFS estimates, RBI and SEBI risk models, pharmaceutical dose studies, credit scorecards and A/B tests in Indian analytics and actuarial roles.

Explained

Framework and vocabulary

Population and parameter: a population is described by a distribution f(x; θ) whose parameter θ (mean μ, variance σ², proportion p, rate λ) is unknown and fixed. The parameter space Θ lists the values θ may take.
Statistic: any function T(X₁,…,Xₙ) of the sample that contains no unknown parameter. An estimator is a statistic used to guess θ; it is a random variable. The number it takes for one actual sample is the estimate.
Sampling distribution: the distribution of the estimator over all possible samples. Its standard deviation is the standard error (SE); SE is the basic measure of precision and falls like 1/√n.
Point estimation reports one value (x̄ = ₹32,400); interval estimation reports a range with a confidence level (₹31,100 to ₹33,700 at 95%). Interval estimation is honest about uncertainty, which is why survey reports publish it.
Notation: θ̂ denotes an estimator of θ. Bias(θ̂) = E(θ̂) − θ.

Properties of a good estimator

Unbiasedness: E(θ̂) = θ for every θ ∈ Θ. The estimator is right "on average" over repeated samples. x̄ is unbiased for μ; s² with divisor (n−1) is unbiased for σ²; the divisor n version is biased downward by factor (n−1)/n.
Consistency: θ̂ₙ → θ in probability as n → ∞. Sufficient condition: Bias → 0 and Var → 0. Consistency is a large-sample property; a biased estimator can still be consistent (e.g. the divisor-n variance).
Efficiency: among unbiased estimators, the one with the smallest variance is most efficient. Relative efficiency of θ̂₁ to θ̂₂ = Var(θ̂₂)/Var(θ̂₁). For a normal population, Var(x̄) = σ²/n while Var(median) ≈ 1.5708σ²/n, so the median's efficiency is about 63.7%.
Sufficiency: T is sufficient for θ if the conditional distribution of the sample given T does not depend on θ - T squeezes out all parameter information. Checked with the Fisher-Neyman factorisation theorem: f(x; θ) = g(T(x); θ)·h(x).
Minimum variance unbiased (MVUE/UMVUE): unbiased with the least variance for all θ. Completeness of the sufficient statistic plus the Lehmann-Scheffé theorem certifies uniqueness.
MSE as a combined criterion: MSE(θ̂) = Var(θ̂) + [Bias(θ̂)]². A slightly biased estimator with a much smaller variance can beat an unbiased one - the bias-variance trade-off used in ridge and shrinkage estimation.
Invariance, robustness, sufficiency of statistics: also asked - the MLE is invariant (if θ̂ is MLE of θ, g(θ̂) is MLE of g(θ)); trimmed means and medians are robust to outliers while x̄ is not.

Cramér-Rao inequality, Fisher information and Rao-Blackwell

Regularity conditions: the support must not depend on θ and differentiation under the integral sign must be valid (fails for U(0, θ), so CRLB does not apply there).
Fisher information I(θ) = E[(∂ log f/∂θ)²] = −E[∂² log f/∂θ²] for one observation; for n iid observations it is nI(θ). More information means a sharper likelihood peak.
Cramér-Rao lower bound (CRLB): Var(θ̂) ≥ 1/[nI(θ)] for any unbiased θ̂. An estimator attaining it is efficient (MVB estimator). For N(μ, σ²) with σ known, Var(x̄) = σ²/n = CRLB exactly, so x̄ is efficient.
Attainment condition: equality holds only if ∂ log L/∂θ = k(θ)[θ̂ − θ]. If no unbiased estimator reaches the bound, the UMVUE may still exist (found via Lehmann-Scheffé, not CRLB).
Rao-Blackwell theorem: if θ̂ is unbiased and T sufficient, then θ* = E(θ̂ | T) is unbiased and Var(θ*) ≤ Var(θ̂) - conditioning on a sufficient statistic never hurts and usually improves. This is the standard recipe for improving a crude unbiased estimator.

Methods of point estimation

Method of moments (MoM): equate as many sample moments to population moments as there are parameters and solve. Simple, needs no distributional derivation, consistent - but can be inefficient and may give impossible values (a negative variance estimate, or p̂ outside [0,1]).
Maximum likelihood estimation (MLE): write the likelihood L(θ) = ∏f(xᵢ; θ), take log, differentiate, set ∂ log L/∂θ = 0, solve, and confirm the second derivative is negative. Where the likelihood is monotone or non-differentiable (uniform, exponential with threshold), maximise it directly by inspection - for U(0,θ) the MLE is max(xᵢ).
- Properties: consistent, asymptotically normal with variance 1/[nI(θ)], asymptotically efficient, invariant, a function of the sufficient statistic - but may be biased in small samples (MLE of σ² in a normal uses divisor n).
Least squares: minimise Σ(observed − fitted)²; gives BLUE estimators in the linear model by the Gauss-Markov theorem without any normality assumption.
Minimum chi-square and minimum variance methods: fit by minimising Σ(Oᵢ − Eᵢ)²/Eᵢ; used for grouped and categorical data.
Bayesian estimation: combine prior π(θ) with likelihood to get the posterior ∝ L(θ)π(θ); report the posterior mean (Bayes estimator under squared-error loss), posterior median (absolute-error loss) or mode (MAP). A credible interval is the Bayesian analogue of a confidence interval and is interpreted as a direct probability about θ.
Choosing between them: MLE for efficiency and large samples; MoM for a quick start or when the likelihood is intractable; least squares for regression; Bayesian when prior information or small samples matter (common in actuarial and clinical work).

Interval estimation

A 100(1−α)% confidence interval is a random interval (L, U) with P(L ≤ θ ≤ U) = 1−α before sampling. After computing it the interval is fixed: the correct reading is that 95 of every 100 such intervals from repeated samples would cover θ, not that θ has a 95% chance of lying in this one.
Pivotal quantity method: find a function of sample and θ whose distribution is free of θ - (x̄ − μ)/(σ/√n) ~ N(0,1), (x̄ − μ)/(s/√n) ~ tₙ₋₁, (n−1)s²/σ² ~ χ²ₙ₋₁ - then invert the probability statement.
Which distribution: σ known (or n large, by CLT) → z; σ unknown and the population roughly normal → t with (n−1) df; variance → χ²; ratio of two variances → F; proportions and large-sample MLEs → z on the asymptotic normal.
Standard critical values: z₀.₀₂₅ = 1.96 (95%), z₀.₀₀₅ = 2.576 (99%), z₀.₀₅ = 1.645 (90%).
Width = 2 × critical value × SE. It widens with higher confidence and larger σ, and narrows as √n grows - halving the width needs four times the sample.
One-sided bounds are used when only a minimum (strength, life) or maximum (impurity, default rate) matters.
Proportions: the Wald interval p̂ ± z√(p̂q̂/n) behaves badly for small n or extreme p̂; the Wilson score and Agresti-Coull (add 2 successes and 2 failures) intervals are preferred and are what good software reports. The finite population correction √((N−n)/(N−1)) applies when the sample exceeds about 5% of the population - standard in NSSO-type survey estimation.
Sample size: fix the margin of error E and invert the width formula. For proportions, p = 0.5 is the conservative choice because p(1−p) is maximum there.

Indian context, standards and practice

Official statistics: MoSPI's NSS, PLFS (quarterly urban and annual reports) and HCES publish estimates with estimated standard errors and relative standard errors; estimation here uses stratified multi-stage designs with survey weights, so SE is computed by design-based or bootstrap methods rather than σ/√n.
Quality control: acceptance sampling and process capability work under the IS 2500 series on sampling inspection (based on ISO 2859/3951); control-chart constants estimate σ from R̄/d₂ rather than s - an estimation choice with a known efficiency loss.
Regulatory use: RBI's expected credit loss and capital models, IRDAI-mandated actuarial reserving, and CDSCO/ICMR clinical trial protocols all require interval estimates and pre-specified sample size calculations; the ICMR National Ethical Guidelines require sample size justification in the protocol.
Exam weight: B.Sc CBCS "Statistical Inference" papers typically devote a full unit to point estimation (properties, CRLB, MLE, MoM, sufficiency) and another to interval estimation; GATE ST and ISS ask CRLB, Rao-Blackwell, Lehmann-Scheffé and MLE derivations directly.

Key points

Estimator vs estimate - the rule (a random variable) vs the number it produces for one sample.
Bias - E(θ̂) − θ; zero bias means correct on average, not correct for your sample.
Standard error - SD of the sampling distribution; σ/√n for the mean, √(pq/n) for a proportion.
Consistency - θ̂ converges in probability to θ; guaranteed if bias → 0 and variance → 0.
Efficiency - smallest variance among unbiased estimators; measured relative to another estimator or to the CRLB.
Sufficiency - all sample information about θ is captured by T; verified by Fisher-Neyman factorisation.
Completeness + sufficiency + unbiasedness - Lehmann-Scheffé gives the unique UMVUE.
Cramér-Rao bound - Var(θ̂) ≥ 1/[nI(θ)]; needs regularity conditions, so it fails for U(0,θ).
Fisher information - I(θ) = −E[∂² log f/∂θ²]; adds across independent observations.
Rao-Blackwellisation - E(θ̂ | sufficient T) is unbiased with variance no larger.
MSE = variance + bias² - the honest single-number criterion; basis of the bias-variance trade-off.
MLE - maximises the likelihood; consistent, asymptotically normal and efficient, invariant, possibly biased in small samples.
Method of moments - equate sample and population moments; simple and consistent but often inefficient.
Gauss-Markov - OLS is BLUE in the linear model under zero-mean, constant-variance, uncorrelated errors; normality not needed.
Confidence interval - estimate ± (critical value × SE); the probability statement belongs to the procedure, not to one computed interval.
t vs z - use t with n−1 df when σ is estimated from a small sample; t is wider and converges to z as n grows.
Width shrinks as √n - a 4× larger sample halves the margin of error.
Bayes estimator - posterior mean under squared-error loss; a credible interval does allow a direct probability statement about θ.

Formulas and examples

Bias and MSE: Bias(θ̂) = E(θ̂) − θ; MSE(θ̂) = E(θ̂ − θ)² = Var(θ̂) + [Bias(θ̂)]² — same units as θ² for MSE.

Sample mean and variance: x̄ = (1/n)Σxᵢ; s² = Σ(xᵢ − x̄)²/(n − 1) (unbiased); SE(x̄) = σ/√n, estimated by s/√n.

Fisher information and CRLB: I(θ) = E[(∂ log f/∂θ)²] = −E[∂² log f/∂θ²]; Var(θ̂) ≥ 1/[n I(θ)] for unbiased θ̂.

Likelihood: L(θ) = ∏ᵢ₌₁ⁿ f(xᵢ; θ); solve ∂ log L/∂θ = 0 with ∂² log L/∂θ² < 0.

CI for μ, σ known: x̄ ± z_{α/2} σ/√n. σ unknown: x̄ ± t_{α/2, n−1} s/√n.

CI for proportion (Wald): p̂ ± z_{α/2} √(p̂(1 − p̂)/n), p̂ = x/n.

CI for σ²: [(n − 1)s² / χ²_{α/2, n−1}, (n − 1)s² / χ²_{1−α/2, n−1}].

Difference of means (σ's unknown, equal, pooled): (x̄₁ − x̄₂) ± t_{α/2, n₁+n₂−2} · s_p√(1/n₁ + 1/n₂), with s_p² = [(n₁−1)s₁² + (n₂−1)s₂²]/(n₁ + n₂ − 2).

Sample size: for a mean, n = (z_{α/2} σ / E)²; for a proportion, n = z²_{α/2} p(1 − p)/E², with E the margin of error in the units of the variable (use p = 0.5 if p is unknown).

Finite population correction: SE(x̄) = (σ/√n)·√((N − n)/(N − 1)), applied when n/N exceeds about 0.05.

Example 1 — Bias, MSE and the (n−1) divisor. A sample of n = 10 from a population with σ² = 25 gives Σ(xᵢ − x̄)² = 198.

Unbiased estimate s² = 198/9 = 22.00.
Biased (divisor n) estimate = 198/10 = 19.80; its expectation is σ²(n−1)/n = 25 × 0.9 = 22.5, so bias = 22.5 − 25 = −2.5.
For a normal population, Var(s²) = 2σ⁴/(n−1) = 2 × 625/9 = 138.9, so MSE(s²) = 138.9.
For the divisor-n version, Var = 2σ⁴(n−1)/n² = 2 × 625 × 9/100 = 112.5, so MSE = 112.5 + (−2.5)² = 118.75 — lower MSE despite the bias, which is exactly the trade-off.

Example 2 — MLE and MoM for the Poisson, and the CRLB. Defects per bale of jute in 50 bales: Σxᵢ = 160.

log L = −nλ + (Σxᵢ) log λ − Σ log(xᵢ!); setting d/dλ = −n + Σxᵢ/λ = 0 gives λ̂ = x̄ = 160/50 = 3.2 defects per bale. MoM gives the same since E(X) = λ.
I(λ) = −E[−x/λ²] = 1/λ, so CRLB = λ/n. Var(x̄) = λ/n — the bound is attained, so x̄ is the efficient (MVB) estimator of λ.
Estimated SE = √(3.2/50) = √0.064 = 0.253; 95% CI for λ: 3.2 ± 1.96 × 0.253 = (2.70, 3.70).

Example 3 — Confidence interval for a mean with σ unknown. Monthly salaries of 16 fresh graduates from one campus: x̄ = ₹32,400, s = ₹4,800.

SE = 4800/√16 = 4800/4 = ₹1,200.
t₀.₀₂₅,₁₅ = 2.131, so margin = 2.131 × 1200 = ₹2,557.
95% CI = 32,400 ± 2,557 = (₹29,843, ₹34,957).
With σ known as ₹4,800 the z interval would be 32,400 ± 1.96 × 1200 = (₹30,048, ₹34,752) — narrower, which is why using z with a small sample understates uncertainty.

Example 4 — Proportion and sample size. In a PLFS-style survey of n = 1,200 urban youth, 384 report being in the labour force.

p̂ = 384/1200 = 0.32; SE = √(0.32 × 0.68/1200) = √0.00018133 = 0.01347.
95% CI = 0.32 ± 1.96 × 0.01347 = 0.32 ± 0.0264 = (0.2936, 0.3464), i.e. about 29.4% to 34.6%.
To cut the margin of error to 2 percentage points: n = 1.96² × 0.32 × 0.68/0.02² = 3.8416 × 0.2176/0.0004 = 2,090 (round up). With p unknown, p = 0.5 gives n = 3.8416 × 0.25/0.0004 = 2,401.

Example 5 — Interval for a variance. Fill weights of 25 packets give s² = 4.84 g² (s = 2.2 g); assume normality.

df = 24; χ²₀.₀₂₅,₂₄ = 39.364 and χ²₀.₉₇₅,₂₄ = 12.401.
(n−1)s² = 24 × 4.84 = 116.16.
95% CI for σ²: (116.16/39.364, 116.16/12.401) = (2.951, 9.367) g², so σ lies in (1.72, 3.06) g — note the interval is asymmetric about s².

Criterion	Meaning	Sample size needed	Typical check
Unbiasedness	Correct on average	Any n	E(θ̂) = θ
Consistency	Converges to θ	Large n	Bias → 0, Var → 0
Efficiency	Least variance	Any n (compare)	Var vs CRLB
Sufficiency	No information lost	Any n	Factorisation theorem

Method	Basis	Strength	Weakness
Method of moments	Match moments	Simple, no derivation	Inefficient; can give impossible values
Maximum likelihood	Maximise L(θ)	Asymptotically efficient, invariant	Small-sample bias; needs the model
Least squares	Minimise Σe²	BLUE, no normality needed	Sensitive to outliers
Bayesian	Posterior ∝ L × prior	Uses prior, small samples	Prior choice is debatable

Where it is used

Official statistics and policy: MoSPI/NSSO, PLFS, HCES and NFHS estimates of unemployment rate, consumption expenditure and health indicators, published with standard errors and design-based intervals.
Banking and finance: RBI-regulated banks and NBFCs estimate PD, LGD and expected credit loss; VaR and volatility estimation in treasury and risk roles at SBI, HDFC Bank, ICICI and rating agencies (CRISIL, ICRA).
Insurance and actuarial: IRDAI-regulated reserving, mortality and claim-frequency estimation, credibility (shrinkage) estimators — core to IAI actuarial exams.
Pharma and clinical research: sample size and confidence intervals for efficacy in CDSCO-approved trials and bioequivalence studies at Sun Pharma, Cipla, Dr Reddy's and CROs.
Manufacturing quality: estimating process mean and σ for capability indices and control limits under the IS 2500 sampling-inspection series; vendor acceptance sampling.
Analytics and market research: A/B test lift estimates with confidence intervals, survey estimation at Nielsen, Kantar and IPSOS, and data-scientist/statistical-analyst roles across Indian product firms.

Interview questions

Q: What is the difference between an estimator and an estimate? A: An estimator is the rule or formula applied to a sample and is a random variable with a sampling distribution; an estimate is the single numerical value it produces for one observed sample.
Q: Why does the sample variance divide by (n−1)? A: Deviations are taken about x̄, not μ, so the sum of squares is systematically too small; dividing by the (n−1) degrees of freedom makes the estimator unbiased for σ².
Q: Can a biased estimator be preferred? A: Yes — if its variance is much smaller, its MSE can be lower; shrinkage and ridge estimators work exactly on this trade-off.
Q: State the Cramér-Rao inequality and one case where it does not apply. A: For any unbiased θ̂ satisfying regularity conditions, Var(θ̂) ≥ 1/[nI(θ)]. It fails for U(0, θ) because the support depends on θ, so differentiation under the integral is invalid.
Q: What does sufficiency mean in plain terms? A: A sufficient statistic carries all the sample's information about the parameter; given its value, the rest of the data tells you nothing more about θ.
Q: How is Rao-Blackwell used? A: Take any unbiased estimator and replace it by its conditional expectation given a sufficient statistic — the result is still unbiased but has variance no larger, and with completeness it is the UMVUE.
Q: Name the properties of the MLE. A: Consistent, asymptotically normal with variance 1/[nI(θ)], asymptotically efficient, invariant under transformation, and a function of the sufficient statistic — but it can be biased in small samples.
Q: When would you use the method of moments over MLE? A: When the likelihood is intractable or has no closed form, or as a starting value for numerical maximisation; it is simpler but generally less efficient.
Q: Interpret a 95% confidence interval correctly. A: The procedure produces intervals that cover the true parameter in 95% of repeated samples; a specific computed interval either contains θ or does not — the 95% is not a probability about that one interval.
Q: When do you use t instead of z? A: When σ is unknown and estimated by s with a small sample from an approximately normal population; use n−1 degrees of freedom, and the two agree as n grows large.
Q: Why is p = 0.5 used in sample size formulas for proportions? A: p(1−p) is maximised at 0.5, so it gives the largest, most conservative sample size when no prior estimate of p exists.
Q: How does a credible interval differ from a confidence interval? A: A credible interval comes from the posterior distribution and permits the direct statement "θ lies in this range with 95% probability"; a confidence interval's coverage refers to the long-run behaviour of the method.

Common mistakes

Saying "there is a 95% probability that μ lies in (30,048, 34,752)" — the probability belongs to the procedure, not to the computed interval.
Using z when σ is unknown and n is small, which gives an interval that is too narrow and overstates precision.
Confusing standard deviation with standard error, or forgetting to divide by √n when computing the margin of error.
Applying the Cramér-Rao bound without checking regularity conditions, then wrongly concluding no better estimator exists for U(0, θ).
Treating unbiasedness as sufficient for a good estimator and ignoring its variance, or treating consistency (a large-n property) as a small-sample guarantee.
Taking the divisor-n MLE of σ² as unbiased, or mixing up s and s² in interval formulas for the variance.
Forgetting the finite population correction in survey work where the sample is a large fraction of the population, and forgetting that the χ² interval for σ² is asymmetric.