Bayesian statistics

In short

Bayesian statistics treats unknown parameters as random variables with probability distributions: a prior belief is combined with observed data through the likelihood to give a posterior belief, using Bayes' theorem. Unlike the frequentist view where a parameter has one fixed true value and probability describes long-run frequencies, the Bayesian view lets you say "there is a 95% probability that the conversion rate lies between 3.1% and 4.4%" — a statement product teams actually want. It is examined in GATE DA (Data Science and AI), GATE CS, AICTE/CBCS papers on Probability and Statistics and Machine Learning, and in nearly every data-science interview via Bayes' theorem puzzles, naive Bayes and A/B-test questions. Indian employers — Flipkart, Swiggy, Zomato, Razorpay, PhonePe, CRED, Dream11, Fractal, Mu Sigma, Tiger Analytics, ISRO/DRDO labs, ICMR epidemiology groups — use it for spam and fraud filters, recommendation, demand forecasting, clinical trials and sequential experimentation.

Explained

Bayes' theorem and its vocabulary

• Conditional probability: P(A|B) = P(A ∩ B)/P(B), for P(B) > 0. Bayes' theorem simply rearranges this to invert the conditioning.
• Law of total probability: if B₁…Bₖ partition the sample space, P(A) = Σᵢ P(A|Bᵢ)P(Bᵢ). This denominator is what makes Bayes' theorem computable.
• Four named pieces for a parameter θ and data D: prior p(θ) (belief before data), likelihood p(D|θ) (how probable the data is for each θ), posterior p(θ|D) (belief after data), evidence / marginal likelihood p(D) = ∫ p(D|θ)p(θ)dθ (a normalising constant in θ).
• Proportional form — the one used in practice: posterior ∝ likelihood × prior. Compute the shape first, normalise later.
• Likelihood is not a probability over θ. p(D|θ) integrated over θ need not equal 1; it is a function of θ for fixed data.
• Sequential updating: today's posterior becomes tomorrow's prior. Updating on D₁ then D₂ gives the same answer as updating on both at once (for conditionally independent data) — the coherence property that makes Bayesian methods natural for streaming data.
• Odds form: posterior odds = prior odds × likelihood ratio. Fast for two-hypothesis exam problems and for medical-test questions.

Priors: types and choice

• Informative prior: encodes real knowledge (past campaign conversion was 3–5%, so Beta(30, 870)).
• Weakly informative prior: keeps estimates in a sane range without asserting much — e.g. Normal(0, 10²) on a regression coefficient. The modern default.
• Non-informative / flat prior: Uniform(0,1) = Beta(1,1) for a probability; Normal with huge variance for a mean. May be improper (does not integrate to 1), which is acceptable only if the posterior is proper.
• Jeffreys prior: p(θ) ∝ √(I(θ)) where I(θ) is Fisher information — invariant under reparameterisation. For a Bernoulli probability it is Beta(0.5, 0.5).
• Conjugate prior: a prior that, with a given likelihood, yields a posterior in the same family — closed-form, no simulation needed. The backbone of exam questions.
• Hierarchical / multilevel prior: parameters of the prior (hyperparameters) themselves get priors, e.g. per-city conversion rates drawn from a common distribution. Gives partial pooling — small cities are shrunk towards the national average.
• Empirical Bayes: hyperparameters estimated from the data itself rather than given a prior; cheap and widely used in industry (IMDb-style weighted ratings, click-through-rate smoothing).
• Sensitivity analysis is expected: report how the posterior changes under two or three reasonable priors. With plenty of data the prior washes out; with little data it dominates.

Conjugate families (the examinable table)

• Beta–Binomial: θ ~ Beta(α, β), x successes in n trials → posterior Beta(α + x, β + n − x). α and β act as "prior successes and failures" (pseudo-counts).
• Normal–Normal (known variance σ²): mean μ ~ Normal(μ₀, τ₀²) → posterior mean is a precision-weighted average of prior mean and sample mean; posterior precision = prior precision + data precision.
• Gamma–Poisson: λ ~ Gamma(α, β) (rate parametrisation), counts x₁…xₙ → posterior Gamma(α + Σxᵢ, β + n). Used for arrivals, defects, claims.
• Inverse-Gamma–Normal: conjugate for an unknown variance σ²; Normal–Inverse-Gamma when both μ and σ² are unknown.
• Dirichlet–Multinomial: the multi-category generalisation of Beta–Binomial; underlies topic models (LDA) and text classification smoothing.
• Conjugacy is a computational convenience, not a requirement. Logistic regression, neural networks and most real models have no conjugate prior, so they need MCMC or variational inference.

Point estimates, intervals and prediction

• Posterior mean E[θ|D] — optimal under squared-error loss; the usual "Bayes estimator".
• Posterior median — optimal under absolute-error loss; robust to skew.
• MAP (maximum a posteriori) — the posterior mode, argmax p(θ|D) = argmax [log p(D|θ) + log p(θ)]. With a flat prior MAP = MLE.
• Regularisation link (asked often): MAP with a Normal(0, τ²) prior on weights = ridge (L2) regression; MAP with a Laplace prior = lasso (L1). Penalty strength λ is a function of prior variance.
• Credible interval: any region holding a stated posterior probability. Equal-tailed uses the 2.5th and 97.5th percentiles; HPD/HDI (highest posterior density) is the shortest such interval and every point inside has higher density than any point outside.
• Credible vs confidence interval: a 95% credible interval is a probability statement about θ given this data; a 95% confidence interval is a statement about the procedure over repeated samples. Do not use the words interchangeably in a viva.
• Posterior predictive distribution: p(x̃|D) = ∫ p(x̃|θ)p(θ|D)dθ — averages over parameter uncertainty instead of plugging in a point estimate, so prediction intervals are honestly wider. Beta–Binomial prior predictive is the Beta-Binomial distribution; Gamma–Poisson gives the negative binomial.
• Posterior predictive checks: simulate replicate datasets from the fitted model and compare their summaries with the observed data to detect misfit.

Hypothesis comparison and decisions

• Bayes factor BF₁₀ = p(D|M₁)/p(D|M₀) — the ratio of marginal likelihoods; multiplies prior odds into posterior odds. Rough reading (Jeffreys/Kass–Raftery scale): 1–3 barely worth mentioning, 3–10 substantial, 10–30 strong, 30–100 very strong, >100 decisive. The scale is a convention, not a rule.
• Unlike a p-value, a Bayes factor can support the null, and it is not inflated by repeated looks at the data — one reason Bayesian A/B tests allow continuous monitoring.
• Lindley's paradox: with a very diffuse prior and large n, a result that a p-value calls "significant" can have a Bayes factor favouring the null. Shows Bayes factors are sensitive to prior width.
• Bayesian decision theory: choose the action a minimising posterior expected loss, E[L(θ, a)|D]. Classification with 0–1 loss reduces to picking the highest posterior class probability; asymmetric business losses (a missed fraud costs ₹20,000, a false block costs ₹200) shift the threshold away from 0.5.
• Model comparison in practice: WAIC and LOO-CV (leave-one-out cross-validation) are preferred over DIC and over raw Bayes factors because marginal likelihoods are hard to estimate and prior-sensitive.
• Bayesian model averaging: weight predictions by posterior model probabilities instead of choosing one model.

Computation: when there is no closed form

• Why it is hard: the evidence p(D) is a high-dimensional integral. Conjugacy avoids it; otherwise approximate.
• Grid approximation: discretise θ, compute likelihood × prior at each point, normalise. Fine for 1–2 parameters; used in teaching and in exam-style numerical problems.
• Rejection and importance sampling: draw from a proposal and reweight; degrades badly in high dimensions.
• Markov Chain Monte Carlo (MCMC): build a Markov chain whose stationary distribution is the posterior, then summarise the draws.
  • Metropolis–Hastings: propose θ*, accept with probability min(1, [p(θ*|D)q(θ|θ*)] / [p(θ|D)q(θ*|θ)]). The unknown p(D) cancels in the ratio — the key insight.
  • Gibbs sampling: cycle through parameters, sampling each from its full conditional; natural for hierarchical and conjugate-block models.
  • Hamiltonian Monte Carlo / NUTS: uses gradients of the log posterior to move efficiently in high dimensions; the engine inside Stan and PyMC.
• Diagnostics expected in an answer: discard burn-in, run several chains, check R̂ (Gelman–Rubin) ≈ 1.00–1.01, check effective sample size (ESS, aim for a few hundred per parameter), inspect trace plots and autocorrelation, watch divergences in HMC.
• Variational inference (VI): replace sampling by optimisation — pick the closest member q of a simple family by maximising the ELBO (equivalently minimising KL(q‖p)). Fast and scalable, but typically underestimates posterior variance.
• Laplace approximation: approximate the posterior by a Normal centred at the MAP with covariance = inverse Hessian of the negative log posterior. Underlies much of Bayesian deep learning.
• Tools: PyMC, Stan (CmdStanPy), NumPyro/JAX, TensorFlow Probability, scikit-learn's BernoulliNB/MultinomialNB/GaussianNB and BayesianRidge, R's brms/rstanarm.

Bayesian models students must know

• Naive Bayes classifier: predict argmax_c P(c)Πⱼ P(xⱼ|c), assuming features are conditionally independent given the class. Variants: multinomial (word counts), Bernoulli (presence/absence), Gaussian (continuous features).
• Laplace (add-one) smoothing: P(word|c) = (count + α)/(total + αV) with α = 1 and V = vocabulary size — a Dirichlet prior in disguise; it prevents a single unseen word from zeroing the whole product. Work with log-probabilities to avoid underflow.
• Bayesian linear regression: Normal prior on coefficients + Normal noise gives a Normal posterior; predictions come with calibrated intervals. Reduces to ridge at the posterior mode.
• Bayesian A/B testing: Beta posteriors for each arm, report P(θ_B > θ_A) and expected loss; supports early stopping and multi-armed bandits (Thompson sampling draws from each arm's posterior and plays the winner) — standard in Indian consumer-app experimentation platforms.
• Hierarchical models: store-level, city-level or doctor-level effects with shrinkage; the usual answer to "how do I model 500 outlets with 20 sales each?"
• Bayesian networks: directed acyclic graphs encoding conditional independence; inference by variable elimination or belief propagation. Used in diagnosis and risk models.
• Gaussian processes, Kalman filters and Bayesian optimisation (Expected Improvement / Upper Confidence Bound for hyperparameter tuning) are all Bayesian updating in continuous spaces.
• Advantages: natural uncertainty quantification, principled inclusion of prior/domain knowledge, works with small samples, sequential and online updating, no multiple-testing inflation from repeated looks.
• Limitations: prior choice is subjective and must be defended, computation is heavy, MCMC can fail silently without diagnostics, regulators and journals still expect frequentist reporting, and results are harder to explain to non-technical stakeholders.

Indian standards and regulatory practice

• Clinical trials: adaptive and Bayesian designs are permitted under the New Drugs and Clinical Trials Rules, 2019 (CDSCO, under the Drugs and Cosmetics Act, 1940) provided the design, priors and stopping rules are pre-specified in the protocol approved by the Ethics Committee and CDSCO; ICMR's National Ethical Guidelines (2017) require pre-registration on CTRI.
• Official statistics: MoSPI and NSSO publish frequentist design-based estimates; Bayesian small-area estimation is used in research work on district-level poverty, nutrition and NFHS indicators rather than in headline releases.
• Banking and insurance: RBI's IRB credit-risk guidance and IRDAI reserving practice accept Bayesian/credibility methods (Bühlmann credibility is empirical Bayes) for low-data segments; model risk management requires documented assumptions and back-testing. Specific circular requirements change — check the current RBI/IRDAI master direction before citing one.
• Exam weight: GATE DA lists Bayes' theorem, conditional probability, and (under machine learning) the naive Bayes classifier and ridge regression — MAP estimation is not named, but ridge is its standard exam link; GATE CS/IT asks Bayes' theorem numericals. Syllabi are revised, so confirm the current year's official syllabus PDF.

Key points

• Bayes' theorem — P(θ|D) = P(D|θ)P(θ)/P(D); inverts conditioning.
• Posterior ∝ likelihood × prior — the working form; the evidence only normalises.
• Prior — belief before data; informative, weakly informative, flat, Jeffreys or hierarchical.
• Likelihood — probability of the observed data as a function of θ; not a distribution over θ.
• Evidence P(D) — marginal likelihood, the integral that makes exact Bayes hard and the denominator in Bayes factors.
• Conjugate prior — prior and posterior in the same family: Beta–Binomial, Gamma–Poisson, Normal–Normal, Dirichlet–Multinomial.
• Pseudo-counts — in Beta(α, β), α−1 prior successes and β−1 prior failures; data swamps them as n grows.
• MAP vs MLE — MAP maximises posterior, MLE maximises likelihood; they coincide under a flat prior.
• MAP = regularisation — Gaussian prior ⇒ ridge/L2, Laplace prior ⇒ lasso/L1.
• Credible interval — a 95% probability statement about θ given the data; HPD is the shortest such interval.
• Confidence interval — a statement about the long-run coverage of a procedure, not about θ.
• Posterior predictive — averages the likelihood over the posterior; wider and better calibrated than plug-in prediction.
• Bayes factor — ratio of marginal likelihoods; converts prior odds to posterior odds and can favour the null.
• MCMC — Metropolis–Hastings, Gibbs, HMC/NUTS; validate with R̂ ≈ 1, ESS, trace plots, burn-in discarded.
• Variational inference — optimisation instead of sampling; fast, but understates uncertainty.
• Naive Bayes — conditional independence of features given the class, plus Laplace smoothing; strong baseline for text.
• Shrinkage — hierarchical models pull small-sample groups towards the overall mean (partial pooling).
• Decision rule — act to minimise posterior expected loss, not merely to maximise probability.

Formulas and examples

Example 1 — Medical screening (the classic base-rate question). A disease affects 0.5% of a population. A test has sensitivity 99% and specificity 95%. A randomly screened person tests positive. What is the probability they have the disease?

• Prior: P(D) = 0.005, P(no D) = 0.995.
• P(+|D) = 0.99 → 0.005 × 0.99 = 0.00495.
• P(+|no D) = 1 − 0.95 = 0.05 → 0.995 × 0.05 = 0.04975.
• P(+) = 0.00495 + 0.04975 = 0.05470.
• P(D|+) = 0.00495 / 0.05470 = 0.0905 ≈ 9.05%.
• In 1,00,000 people: 500 diseased (495 test positive), 99,500 healthy (4,975 false positives) → 495/5,470 = 9.05%. A second independent positive test gives prior odds 495:4,975 × (0.99/0.05) = 490.05:248.75 → 66.3%.

Example 2 — Beta–Binomial for an app conversion rate. Last quarter's checkout conversion was around 4%, so take a prior Beta(8, 192) (mean 8/200 = 4%, worth 200 pseudo-trials). A new UI gets 30 conversions in 500 sessions.

• Posterior: Beta(8 + 30, 192 + 500 − 30) = Beta(38, 662).
• Posterior mean = 38/700 = 0.05429 (5.43%); MLE alone = 30/500 = 6.00%; prior mean = 4% — the posterior sits between, nearer the data.
• Posterior SD = √[(0.05429 × 0.94571)/701] = √(0.00007324) = 0.00856 (0.86 pp).
• Approximate 95% credible interval = 0.05429 ± 1.96 × 0.00856 = (3.75%, 7.11%).
• Posterior mode = (38−1)/(700−2) = 37/698 = 5.30% (the MAP estimate).

Example 3 — Normal–Normal update on a measured mean. A sensor's bias is believed to be μ₀ = 2.0 mm with prior SD τ₀ = 1.0 mm. Nine calibration readings average x̄ = 5.0 mm, with known measurement SD σ = 3.0 mm.

• Prior precision = 1/1.0² = 1.000; data precision = n/σ² = 9/9 = 1.000; total = 2.000 → τₙ² = 0.5, τₙ = 0.7071 mm.
• μₙ = (2.0 × 1.000 + 5.0 × 1.000)/2.000 = 3.50 mm.
• 95% credible interval = 3.50 ± 1.96 × 0.7071 = (2.11 mm, 4.89 mm).
• Weight on the data = 1.000/2.000 = 50%: with 9 readings of SD 3 mm, the data is exactly as informative as the prior. With n = 36, data precision = 4, posterior mean = (2 + 20)/5 = 4.40 mm — the prior fades.

Example 4 — Naive Bayes spam filter with Laplace smoothing. Training set: 60 spam, 40 ham. Word "loan" appears 24 times in spam (total 1,200 spam word tokens) and 2 times in ham (total 800 ham tokens). Vocabulary V = 500. Classify a one-word message "loan" (α = 1).

• P(spam) = 0.60, P(ham) = 0.40.
• P(loan|spam) = (24 + 1)/(1,200 + 500) = 25/1,700 = 0.014706.
• P(loan|ham) = (2 + 1)/(800 + 500) = 3/1,300 = 0.0023077.
• Spam score = 0.60 × 0.014706 = 0.0088235; ham score = 0.40 × 0.0023077 = 0.00092308.
• P(spam|"loan") = 0.0088235/(0.0088235 + 0.00092308) = 0.9053 → classify as spam.
• Without smoothing a word unseen in ham would give P = 0, forcing P(spam) = 1 regardless of other evidence.

Example 5 — Gamma–Poisson for support-ticket arrivals. A helpdesk historically receives about 20 tickets/hour; encode this as Gamma(α = 40, β = 2) (mean 40/2 = 20, equivalent to 2 hours of past data). Over the next 5 hours, 120 tickets arrive.

• Posterior: Gamma(40 + 120, 2 + 5) = Gamma(160, 7).
• Posterior mean rate = 160/7 = 22.857 tickets/hour; posterior SD = √160/7 = 12.649/7 = 1.807.
• Approximate 95% credible interval = 22.857 ± 1.96 × 1.807 = (19.32, 26.40) tickets/hour.
• Posterior predictive for the next hour is negative binomial with mean 22.857 and variance 22.857 × (1 + 1/7) = 26.12 — wider than a plug-in Poisson, which would claim variance 22.857.

# Beta-Binomial update and a 95% equal-tailed credible interval (not the HDI)
from scipy.stats import beta
a0, b0 = 8, 192            # prior: ~4% conversion, 200 pseudo-trials
x, n   = 30, 500           # observed conversions / sessions
a, b   = a0 + x, b0 + n - x            # Beta(38, 662)
print(a/(a+b), beta.ppf([0.025, 0.975], a, b))
# P(variant B beats A) by Monte Carlo
import numpy as np
A = beta.rvs(8+22, 192+480-22, size=200_000)
B = beta.rvs(a, b, size=200_000)
print("P(B > A) =", (B > A).mean())

Where it is used

• Consumer internet and fintech: Bayesian A/B tests and Thompson-sampling bandits for pricing, ranking and notification timing at Flipkart, Swiggy, Zomato, Myntra, Dream11; fraud and risk scoring at Razorpay, PhonePe, Paytm, CRED, where asymmetric loss sets the block threshold.
• Analytics and consulting roles: Data Scientist, Decision Scientist, Quantitative Analyst at Fractal, Mu Sigma, Tiger Analytics, ZS Associates, Quantiphi — hierarchical demand and marketing-mix models with shrinkage across SKUs, stores and cities.
• Healthcare and pharma: Bayesian adaptive designs and interim analyses in CDSCO-regulated trials (Serum Institute, Bharat Biotech, Dr Reddy's, Biocon); ICMR and NIE epidemiology groups use Bayesian hierarchical models for disease mapping and small-area prevalence.
• Banking, insurance and actuarial work: credibility (empirical Bayes) premium rating under IRDAI, PD/LGD estimation for thin portfolios under RBI's IRB framework, operational-risk scenario analysis.
• Aerospace, defence and sensing: Kalman and particle filters — recursive Bayesian estimation — in ISRO launch-vehicle and NavIC/GNSS navigation, DRDO tracking, and robotics SLAM.
• Core ML engineering: spam/abuse filters (naive Bayes baselines), Bayesian optimisation for hyperparameter tuning, uncertainty estimates for safety-critical and active-learning systems, probabilistic forecasting of energy demand and inventory.

Interview questions

• Q: State Bayes' theorem and name each term. A: P(θ|D) = P(D|θ)P(θ)/P(D), where P(θ) is the prior, P(D|θ) the likelihood, P(D) the marginal likelihood or evidence, and P(θ|D) the posterior.
• Q: A disease has prevalence 0.5% and a test is 99% sensitive, 95% specific — a positive result means what? A: About 9%; false positives from the large healthy group (4,975 per lakh) swamp the 495 true positives, so a low base rate makes a positive test weak evidence.
• Q: Difference between a 95% confidence interval and a 95% credible interval? A: The credible interval says there is 95% posterior probability that θ lies inside, given this data and prior; the confidence interval says 95% of intervals built this way would contain the fixed θ over repeated sampling.
• Q: What is a conjugate prior and why is it useful? A: A prior whose posterior belongs to the same family under the given likelihood (e.g. Beta with Binomial), giving a closed-form posterior with no numerical integration or sampling.
• Q: How is MAP estimation related to regularised MLE? A: MAP maximises log-likelihood plus log-prior, so a zero-mean Gaussian prior produces the ridge/L2 penalty and a Laplace prior the lasso/L1 penalty; a flat prior reduces MAP to MLE.
• Q: Why is naive Bayes "naive", and why does it still work? A: It assumes features are conditionally independent given the class, which is usually false; but the argmax of the class score is often still correct even when the probabilities themselves are poorly calibrated.
• Q: What breaks if you skip Laplace smoothing in a text classifier? A: Any word unseen for a class gives zero likelihood, and one zero factor kills the whole product, so the class is ruled out regardless of all other evidence.
• Q: Why can you peek at a Bayesian A/B test but not a frequentist one? A: The posterior is a statement about the current data only, so updating repeatedly is coherent; repeated frequentist significance tests inflate the Type I error rate unless you use alpha-spending or sequential corrections.
• Q: How does Metropolis–Hastings avoid computing the evidence? A: The acceptance ratio uses p(θ*|D)/p(θ|D), and the unknown normalising constant p(D) is identical in numerator and denominator, so it cancels.
• Q: How do you know an MCMC run has converged? A: Multiple chains with R̂ close to 1 (≤1.01), adequate effective sample size, stationary well-mixed trace plots, low autocorrelation after burn-in, and no divergent transitions in HMC.
• Q: What is shrinkage in a hierarchical model, and when do you want it? A: Group estimates are pulled towards the population mean in inverse proportion to their sample size — ideal when you have many groups with few observations each, such as 500 stores with 20 sales apiece.
• Q: If a reviewer objects that your prior is subjective, what do you do? A: Report a sensitivity analysis across a weakly informative, a flat and an informative prior, show that the conclusion is stable (or say honestly where it is not), and document the prior's justification before seeing the data.

Common mistakes

• Inverting the conditional: treating P(D|θ) as P(θ|D) — the prosecutor's fallacy; a 99% sensitive test does not give a 99% chance of disease.
• Ignoring the base rate, or forgetting the false-positive branch in the denominator of the total probability law.
• Describing a confidence interval as "95% probability that θ is in this range" — that is the credible interval's meaning.
• Picking a flat prior as if it were assumption-free: Uniform on θ is not uniform on log θ or on odds, and improper priors can give an improper posterior.
• Forgetting the +1 offsets in the Beta posterior mode formula, (α+x−1)/(α+β+n−2), or quoting the mean as the MAP.
• Multiplying many small probabilities in naive Bayes instead of summing logs — numerical underflow; and dropping the class prior log P(c) altogether.
• Reporting MCMC summaries without diagnostics, or including burn-in draws and a single short chain in the posterior.
• Using a plug-in point estimate for prediction instead of the posterior predictive, so the stated prediction interval is too narrow.

Related topics

Probability distributions · Statistical inference and estimation · Hypothesis testing · Experiment design and A/B testing · Causal inference basics · Optimisation for machine learning