Hypothesis testing

In short

Hypothesis testing is a formal procedure for deciding whether sample data provide enough evidence to reject a default claim (the null hypothesis) about a population or a model. It converts a business or scientific question into a test statistic, compares that statistic against a reference distribution, and reports a decision with a quantified error risk. It is examined in the GATE Data Science and AI paper, in university AI/DS statistics papers (AICTE model curriculum, UGC CBCS), and in CAT/placement aptitude rounds. In industry it underpins A/B tests at Flipkart, Swiggy, Zomato and PhonePe, clinical trial analysis under CDSCO/ICMR rules, model-comparison claims in ML, and quality acceptance sampling in Indian manufacturing.

Explained

Framework and definitions

• Statistical hypothesis: a statement about a population parameter (θ) or distribution. Null hypothesis H₀ is the status quo / no-effect claim, always containing an equality; alternative H₁ (or Hₐ) is what we hope to establish.
• Simple vs composite: simple hypothesis fixes the distribution completely (H₀: μ = 50, σ known); composite leaves a range (H₁: μ > 50).
• Test statistic: a function of the sample whose distribution under H₀ is known (z, t, χ², F).
• Critical region (rejection region) R: set of statistic values for which H₀ is rejected. Critical value: its boundary.
• Level of significance α: P(reject H₀ | H₀ true) — the Type I error probability we are willing to tolerate; conventionally 0.05, 0.01 or 0.10, fixed *before* seeing data.
• p-value: probability, computed under H₀, of a test statistic as extreme or more extreme than the observed one. Reject H₀ if p ≤ α. It is not P(H₀ true) and not the probability the result is due to chance alone.
• One-tailed vs two-tailed: direction in H₁ decides. H₁: μ ≠ μ₀ → two-tailed, α split as α/2 in each tail; H₁: μ > μ₀ → right-tailed; H₁: μ < μ₀ → left-tailed. A one-tailed test has more power for the same α but can detect only one direction.

Errors, power and the Neyman–Pearson view

• Type I error (α): rejecting a true H₀ — a false alarm. Type II error (β): failing to reject a false H₀ — a miss.
• Power = 1 − β: probability of correctly rejecting H₀ at a specified true parameter value. Power rises with larger n, larger effect size, smaller σ, and larger α.
• Trade-off: for fixed n, lowering α raises β. Only increasing n (or reducing variance by better design) lowers both.
• Power function / OC curve: power plotted against the true parameter; the operating characteristic curve (1 − power) is used in acceptance sampling. Producer's risk = α, consumer's risk = β in sampling inspection (IS 2500 series on sampling inspection by attributes).
• Neyman–Pearson lemma: among all tests of size α for a simple H₀ vs simple H₁, the likelihood-ratio test that rejects when L(x|θ₁)/L(x|θ₀) > k is most powerful. Basis of UMP tests for monotone likelihood-ratio families.
• Likelihood ratio test (LRT), general: λ = sup_{θ∈Θ₀} L(θ) / sup_{θ∈Θ} L(θ); under regularity, −2 ln λ → χ²_r where r = number of restrictions (Wilks' theorem). Wald and score (Lagrange multiplier) tests are asymptotically equivalent alternatives.
• Effect size: standardised magnitude, e.g. Cohen's d = (x̄ − μ₀)/s. Statistical significance without a meaningful effect size is practically useless; with large n, trivial differences become significant.

Step-by-step procedure (exam-standard)

1. State H₀ and H₁ in terms of a parameter; decide one- or two-tailed.
2. Fix α and check assumptions (random sampling, independence, normality or large n, variance known/unknown, equal variances).
3. Choose the test statistic and its null distribution, with degrees of freedom.
4. Compute the critical value(s) from tables, or compute the p-value.
5. Compute the statistic from the sample.
6. Decide: reject H₀ if the statistic falls in the critical region (or p ≤ α).
7. State the conclusion in the language of the problem, with the confidence interval and effect size.

Parametric tests for means, proportions and variances

• z-test for a mean: σ known, or n ≥ 30 (CLT). Statistic z = (x̄ − μ₀)/(σ/√n).
• One-sample t-test: σ unknown, small n, population approximately normal; df = n − 1.
• Two-sample independent t-test (pooled): assumes equal variances; df = n₁ + n₂ − 2. If variances unequal, use Welch's t-test with approximate df (Satterthwaite).
• Paired t-test: for matched pairs (before/after on the same subject); apply a one-sample t-test to the differences, df = n − 1. Removes between-subject variability, so it is more powerful than an unpaired test on the same data.
• z-test for proportions: one-sample uses p₀ in the standard error; two-sample uses the pooled proportion p̂ = (x₁ + x₂)/(n₁ + n₂).
• χ² test for variance: (n − 1)s²/σ₀², df = n − 1, for a normal population.
• F-test for equality of two variances: F = s₁²/s₂² (larger over smaller), df = (n₁ − 1, n₂ − 1). Very sensitive to non-normality; Levene's or Bartlett's test is preferred in practice.

Chi-square, ANOVA and non-parametric tests

• χ² goodness of fit: χ² = Σ(Oᵢ − Eᵢ)²/Eᵢ, df = k − 1 − (number of parameters estimated). Requires Eᵢ ≥ 5; pool small cells.
• χ² test of independence: r × c contingency table, Eᵢⱼ = (row total × column total)/grand total, df = (r − 1)(c − 1). Yates' correction subtracts 0.5 from |O − E| in a 2 × 2 table. Use Fisher's exact test when expected counts are small.
• One-way ANOVA: H₀: μ₁ = μ₂ = … = μ_k. Partition SST = SSB + SSW; F = MSB/MSW with df (k − 1, N − k). Assumes normality, independence and homoscedasticity. A significant F says "not all equal", so follow with post-hoc tests (Tukey HSD, Bonferroni).
• Two-way ANOVA: tests two factors and their interaction; used in designed experiments.
• Non-parametric alternatives (distribution-free): sign test and Wilcoxon signed-rank (paired t), Mann–Whitney U / Wilcoxon rank-sum (two-sample t), Kruskal–Wallis H (one-way ANOVA), Friedman test (repeated measures), Kolmogorov–Smirnov and Shapiro–Wilk for normality, runs test for randomness. Lower power than parametric tests when normality holds, but valid for ordinal data, outliers and small samples.
• Permutation and bootstrap tests: build the null distribution by resampling; assumption-light and standard in modern ML evaluation.

Multiple testing, practical issues and misuse

• Multiplicity: testing m hypotheses at α each inflates the family-wise error rate to ≈ 1 − (1 − α)^m. For m = 20 at α = 0.05 that is about 64%.
• Bonferroni correction: use α/m per test — simple, conservative. Holm–Bonferroni is uniformly better. Benjamini–Hochberg controls the false discovery rate (FDR) and is standard in genomics and large-scale A/B programmes.
• p-hacking / HARKing / optional stopping: peeking at an A/B test and stopping on significance badly inflates Type I error; use sequential designs (SPRT, alpha-spending, group-sequential bounds) if you must look early.
• Confidence interval duality: a two-tailed test at level α rejects H₀: θ = θ₀ exactly when θ₀ lies outside the (1 − α) confidence interval. Reporting the interval conveys magnitude and uncertainty, not just a yes/no.
• "Fail to reject" ≠ "H₀ is true": absence of evidence, often just low power. Equivalence testing (TOST) is the correct tool for proving "no meaningful difference" — the basis of bioequivalence studies accepted by CDSCO under the New Drugs and Clinical Trials Rules, 2019.
• Indian practice notes: clinical trials follow ICMR National Ethical Guidelines (2017) and CDSCO rules with pre-registered protocols on CTRI; official statistics (NSO/MoSPI surveys such as PLFS and HCES) report standard errors so that differences can be tested; BIS acceptance sampling standards fix α and β as producer's and consumer's risks.
• Bayesian contrast: frequentist testing conditions on H₀ and controls long-run error rates; Bayesian testing reports posterior probabilities and Bayes factors, which answer "how probable is H₁?" directly.

Key points

• H₀ - default no-effect claim with equality; only H₀ is ever tested, and evidence can reject it but never prove it.
• H₁ - the research claim; its form (≠, >, <) fixes whether the test is two- or one-tailed.
• α - probability of a Type I error, chosen before the data; 0.05 is convention, not law.
• β and power - β is the miss rate; power = 1 − β, and 0.80 is the usual minimum target in design.
• p-value - P(data this extreme or more | H₀); reject when p ≤ α. Not the probability H₀ is true.
• Critical region - the set of statistic values leading to rejection; its total probability under H₀ equals α.
• z vs t - z when σ is known or n is large; t with df = n − 1 when σ is estimated from a small sample.
• Paired vs unpaired - pairing removes subject-to-subject variance and gives higher power for the same n.
• χ² - used for variance of a normal population, goodness of fit, and independence in contingency tables.
• F-test - ratio of two variances; also the ANOVA statistic MSB/MSW.
• ANOVA - one omnibus test for k means, avoiding the multiplicity of pairwise t-tests; needs post-hoc tests after rejection.
• Neyman–Pearson lemma - the likelihood-ratio test is the most powerful test of a simple null against a simple alternative at level α.
• Wilks' theorem - −2 ln(likelihood ratio) is asymptotically χ² with df equal to the number of constraints.
• Non-parametric tests - Mann–Whitney, Wilcoxon, Kruskal–Wallis; valid without normality, slightly less powerful when normality holds.
• Multiple comparisons - Bonferroni (α/m) controls family-wise error; Benjamini–Hochberg controls the false discovery rate.
• CI–test duality - H₀: θ = θ₀ is rejected at level α iff θ₀ falls outside the (1 − α) confidence interval.
• Effect size - significance says "an effect exists"; effect size says whether it matters. Report both.
• Sample size - driven by α, desired power, effect size and σ; n scales roughly as 1/(effect size)².

Formulas and examples

Example 1 — One-sample z-test (manufacturing tolerance). A Pune auto-component plant claims the mean length of a shaft is 50.0 mm with σ = 0.40 mm. A sample of n = 64 gives x̄ = 50.12 mm. Test at α = 0.05 whether the mean has shifted (two-tailed).

• SE = 0.40/√64 = 0.40/8 = 0.05 mm.
• z = (50.12 − 50.00)/0.05 = +2.40.
• Critical values ±1.96; |2.40| > 1.96 → reject H₀. p-value = 2(1 − Φ(2.40)) = 2(0.0082) = 0.0164 < 0.05.
• 95% CI: 50.12 ± 1.96(0.05) = (50.022, 50.218) mm — excludes 50.0, agreeing with the test.
• Conclusion: the process mean has shifted upward by about 0.12 mm; re-centre the machine setting.

Example 2 — Paired t-test (training effectiveness). A BPO measures calls handled per hour for 8 agents before and after training. Differences (after − before): 4, 2, 5, −1, 3, 6, 2, 3.

• Σd = 24 → d̄ = 3.0. Deviations from mean: 1, −1, 2, −4, 0, 3, −1, 0 → squares 1, 1, 4, 16, 0, 9, 1, 0; Σ = 32.
• s_d² = 32/7 = 4.571 → s_d = 2.138. SE = 2.138/√8 = 2.138/2.828 = 0.756.
• t = 3.0/0.756 = 3.97, df = 7. One-tailed critical t₀.₀₅,₇ = 1.895; two-tailed t₀.₀₂₅,₇ = 2.365. Either way, reject H₀: training improved throughput.
• 95% CI for the mean gain: 3.0 ± 2.365(0.756) = (1.21, 4.79) calls/hour. Cohen's d = 3.0/2.138 = 1.40, a large effect.

Example 3 — Two-proportion z-test (A/B test on checkout). An Indian e-commerce app tests a new checkout page. Control: 1,200 users, 96 conversions (p̂₁ = 0.080). Variant: 1,200 users, 132 conversions (p̂₂ = 0.110). Test at α = 0.05, two-tailed.

• Pooled p̂ = (96 + 132)/2,400 = 228/2,400 = 0.095; p̂(1 − p̂) = 0.095 × 0.905 = 0.085975.
• SE = √[0.085975 × (1/1200 + 1/1200)] = √[0.085975 × 0.0016667] = √0.00014329 = 0.011970.
• z = (0.110 − 0.080)/0.011970 = 2.506. |z| > 1.96 → reject H₀; p-value ≈ 2(1 − Φ(2.51)) ≈ 0.0122.
• Relative lift = 0.030/0.080 = 37.5%. If each conversion yields ₹450 gross margin, 1,200 extra visitors a day give 36 extra conversions a day ≈ ₹16,200 a day.
• Design check: to detect 0.080 → 0.110 at α = 0.05, power 0.80: n = (1.96 + 0.8416)² [0.08(0.92) + 0.11(0.89)]/(0.03)² = 7.849 × (0.0736 + 0.0979)/0.0009 = 7.849 × 0.1715/0.0009 ≈ 1,496 per arm — so 1,200 per arm was slightly underpowered; the significant result here is still valid, but a replication is prudent.

Example 4 — Chi-square test of independence. 400 final-year students are classified by stream and whether they were placed.

• Expected placed = row total × 320/400 = 0.8 × row total → 120, 120, 80; expected not placed → 30, 30, 20.
• χ² = (120−120)²/120 + (30−30)²/30 + (130−120)²/120 + (20−30)²/30 + (70−80)²/80 + (30−20)²/20
• = 0 + 0 + 100/120 + 100/30 + 100/80 + 100/20 = 0.8333 + 3.3333 + 1.25 + 5.0 = 10.42.
• df = (3 − 1)(2 − 1) = 2; χ²₀.₀₅,₂ = 5.991. 10.42 > 5.991 → reject H₀: placement rate depends on stream (p ≈ 0.0055).

Example 5 — One-way ANOVA (three model variants). Accuracy (%) of a classifier on 4 folds each: A: 82, 84, 86, 88 (mean 85); B: 88, 90, 90, 92 (mean 90); C: 84, 86, 88, 90 (mean 87). Grand mean = (85 + 90 + 87)/3 = 87.333.

• SSB = 4[(85 − 87.333)² + (90 − 87.333)² + (87 − 87.333)²] = 4[5.4444 + 7.1111 + 0.1111] = 4 × 12.6667 = 50.667; df = 2 → MSB = 25.333.
• SSW: A deviations −3,−1,1,3 → 20; B −2,0,0,2 → 8; C −3,−1,1,3 → 20. SSW = 48; df = 9 → MSW = 5.333.
• F = 25.333/5.333 = 4.75; F₀.₀₅(2,9) = 4.26 → reject H₀ at 5% (but not at 1%, where F = 8.02). Follow with Tukey HSD; the A-vs-B gap of 5 points is the likely driver.

from scipy import stats
import numpy as np

# Example 3 reproduced: two-proportion z-test
x1, n1, x2, n2 = 96, 1200, 132, 1200
p1, p2 = x1/n1, x2/n2
p  = (x1 + x2)/(n1 + n2)
se = np.sqrt(p*(1-p)*(1/n1 + 1/n2))
z  = (p2 - p1)/se
print(z, 2*(1 - stats.norm.cdf(abs(z))))     # 2.506, 0.0122

# Paired t-test (Example 2) and chi-square (Example 4)
d = np.array([4, 2, 5, -1, 3, 6, 2, 3])
print(stats.ttest_1samp(d, 0))               # t = 3.97, df = 7
obs = np.array([[120, 30], [130, 20], [70, 30]])
print(stats.chi2_contingency(obs, correction=False))   # chi2 = 10.42, df = 2

Where it is used

• Consumer internet and fintech: A/B and multivariate testing of UI, pricing and notifications at Flipkart, Myntra, Swiggy, Zomato, PhonePe, Paytm; data-scientist and growth-analyst roles run two-proportion and sequential tests daily.
• Pharma and clinical research: Sun Pharma, Cipla, Dr. Reddy's, Biocon and CROs run superiority, non-inferiority and bioequivalence (TOST) tests under CDSCO's New Drugs and Clinical Trials Rules, 2019, with protocols registered on CTRI and ethics review per ICMR guidelines.
• Manufacturing quality and Six Sigma: control-chart out-of-control signals, process-capability claims, acceptance sampling with producer's and consumer's risks (BIS sampling-inspection standards) at Tata Motors, Bosch, Maruti suppliers; roles in quality engineering and reliability.
• Official statistics and policy: MoSPI/NSO surveys (PLFS, HCES) and the MoHFW/IIPS NFHS publish standard errors so that year-on-year or state-wise differences can be tested; economists at RBI, NITI Aayog and NCAER test policy effects.
• Machine learning engineering: comparing models across cross-validation folds (paired t, 5×2 CV, McNemar's test for classifier errors), drift detection using KS or chi-square tests, and guardrail metrics in online experiments.
• Agriculture and public health trials: ICAR field trials use ANOVA on randomised block designs; ICMR and state health departments test intervention effects in district-level programmes.

Interview questions

• Q: What exactly does a p-value of 0.03 mean? A: If H₀ were true, there is a 3% chance of observing data at least as extreme as ours. It is not the probability that H₀ is true nor the probability the finding is a fluke.
• Q: Why can we never "accept" the null hypothesis? A: Failing to reject only means the evidence was insufficient at the chosen α, which may be due to low power; to claim no meaningful difference you need an equivalence test with a pre-set margin.
• Q: When do you use a z-test instead of a t-test? A: When the population σ is known, or n is large enough (typically ≥ 30) that s estimates σ well and the t distribution is close to normal.
• Q: Explain Type I vs Type II error with a business cost. A: Type I ships a useless feature (wasted engineering, possible revenue loss); Type II discards a genuinely better feature. Set α and power from the relative cost of each.
• Q: How does sample size affect significance? A: Standard error falls as 1/√n, so with large n even trivially small effects become statistically significant — always report the effect size and confidence interval.
• Q: Why is peeking at an A/B test every day a problem? A: Repeated testing on accumulating data multiplies opportunities to cross the threshold, so the real Type I error far exceeds α; use sequential or alpha-spending methods instead.
• Q: 20 metrics, α = 0.05 each — what is the chance of at least one false positive? A: About 1 − 0.95²⁰ ≈ 64%. Control it with Bonferroni (α/20) for family-wise error or Benjamini–Hochberg for FDR.
• Q: State the Neyman–Pearson lemma. A: For a simple null against a simple alternative, the test that rejects when the likelihood ratio exceeds a constant chosen to give size α is the most powerful test at that level.
• Q: Difference between paired and unpaired t-tests? A: Paired tests analyse within-subject differences, eliminating between-subject variability and giving higher power with df = n − 1; unpaired tests compare independent group means with df = n₁ + n₂ − 2.
• Q: Why ANOVA instead of many t-tests? A: Three groups need three pairwise tests, inflating family-wise error to about 14% at α = 0.05; ANOVA tests all means in one F-test at the stated α, with post-hoc comparisons afterwards.
• Q: What does a chi-square test of independence assume? A: Independent observations, counts (not percentages), and expected cell frequencies of at least 5 — otherwise pool categories or use Fisher's exact test.
• Q: How do a confidence interval and a two-tailed test relate? A: They are equivalent: H₀: θ = θ₀ is rejected at level α precisely when θ₀ lies outside the (1 − α) interval, and the interval additionally shows the size of the effect.

Common mistakes

• Treating the p-value as P(H₀ true), or reading p = 0.06 as "proof of no effect" rather than insufficient evidence.
• Choosing a one-tailed test after seeing the data's direction, or halving a two-tailed p-value post hoc — this doubles the true Type I error.
• Using z instead of t for small samples with unknown σ, or forgetting that s uses the divisor (n − 1).
• Wrong degrees of freedom: n instead of n − 1, or (r − 1)(c − 1) miscounted in contingency tables; using percentages instead of raw counts in a χ² test.
• Running an unpaired test on paired data (before/after on the same subjects), which throws away power and inflates the standard error.
• Ignoring assumptions — normality, independence, equal variances — and not checking them with Shapiro–Wilk or Levene's test, or switching to Welch's t / a non-parametric test.
• Reporting significance without effect size, confidence interval, or any correction for the many metrics and variants tested.

Related topics

Statistical inference and estimation · Probability distributions · Experiment design and A/B testing · Bayesian statistics · Descriptive statistics · Causal inference basics