Probability distributions

In short

A probability distribution is the rule that assigns probability to the values a random variable can take - a PMF for discrete variables, a PDF for continuous ones, and a CDF for both. It is the mathematical core of AI and data science: every loss function, every generative model, every noise assumption and every confidence statement rests on an assumed distribution. Indian university syllabi (AICTE model curriculum for AI&DS, UGC CBCS "Probability and Statistics") examine definitions, standard distributions with their means, variances and MGFs, and the Central Limit Theorem; GATE DA (Data Science and AI), GATE ST and CSIR/UGC-NET set numerical questions on exactly these. In interviews at Indian firms - TCS Research, Infosys, Wipro, Fractal, Mu Sigma, Flipkart, Swiggy, ZS Associates, banks and insurers - candidates are asked to pick the right distribution for a described situation and justify it.

Explained

Random variables, PMF, PDF, CDF

• A random variable X is a function from the sample space to the real line. Discrete if its range is countable (dice, counts); continuous if it takes values over an interval (time, voltage, weight).
• PMF (discrete): p(x) = P(X = x), with p(x) ≥ 0 and Σ p(x) = 1. P(X = x) is a genuine probability.
• PDF (continuous): f(x) ≥ 0 with ∫f(x)dx = 1 over the support. f(x) is a density, not a probability - P(X = x) = 0 for every single point; only intervals have probability.
• CDF: F(x) = P(X ≤ x). Non-decreasing, right-continuous, F(−∞) = 0, F(∞) = 1. For continuous X, f(x) = dF/dx; P(a < X ≤ b) = F(b) − F(a).
• Quantile function F⁻¹(u) inverts the CDF - the basis of inverse-transform sampling and of QQ plots.
• Mixed distributions exist (e.g. insurance claim = 0 with probability 0.9, continuous above) - handled by a CDF with a jump.

Expectation, variance and moments

• Expectation: E[X] = Σ x p(x) or ∫ x f(x) dx - the long-run average, the "centre of mass" of the distribution.
• Linearity: E[aX + bY + c] = aE[X] + bE[Y] + c, always, whether or not X and Y are independent.
• Variance: Var(X) = E[(X − μ)²] = E[X²] − (E[X])². Var(aX + b) = a²Var(X). Standard deviation σ = √Var.
• Moments: k-th raw moment E[Xᵏ]; central moment E[(X−μ)ᵏ]. Skewness = μ₃/σ³ (asymmetry), kurtosis = μ₄/σ⁴ (tail weight; normal = 3, so excess kurtosis = 0).
• MGF: M(t) = E[e^{tX}]. M'(0) = E[X], M''(0) = E[X²]. MGFs are unique to a distribution and multiply for independent sums - the standard exam tool for proving that a sum of independent Poissons is Poisson, or of independent normals is normal.
• Characteristic function φ(t) = E[e^{itX}] always exists (used where the MGF does not, e.g. Cauchy).
• Inequalities: Markov P(X ≥ a) ≤ E[X]/a for X ≥ 0; Chebyshev P(|X−μ| ≥ kσ) ≤ 1/k² - distribution-free bounds, examined often.

Standard discrete distributions

• Bernoulli(p): one trial, X ∈ {0,1}. Mean p, variance p(1−p) - maximised at p = 0.5. Models a single click, default, pass/fail; the likelihood behind logistic regression.
• Binomial(n, p): number of successes in n *independent*, identical trials with constant p. Mean np, variance np(1−p). Sampling with replacement.
• Hypergeometric(N, K, n): successes when sampling without replacement from a finite lot of N with K defectives - the correct model for batch inspection; tends to binomial when n ≪ N.
• Geometric(p): number of trials up to and including the first success, mean 1/p, variance (1−p)/p². The only discrete memoryless distribution.
• Negative binomial(r, p): trials until the r-th success; also used for over-dispersed counts (variance > mean) where Poisson fails.
• Poisson(λ): counts of rare independent events in a fixed interval; mean = variance = λ. Limit of Binomial as n → ∞, p → 0 with np = λ (rule of thumb: n ≥ 20, p ≤ 0.05).
• Discrete uniform: equally likely outcomes, mean (n+1)/2 for 1…n, variance (n²−1)/12.
• Multinomial: extends binomial to k categories - the softmax/cross-entropy model in classification.

Standard continuous distributions

• Uniform(a,b): f = 1/(b−a); mean (a+b)/2, variance (b−a)²/12. Used for random initialisation and inverse-transform sampling.
• Exponential(λ): waiting time to the next Poisson event; f = λe^{−λx}, mean 1/λ, variance 1/λ². Memoryless: P(X > s+t | X > s) = P(X > t) - the continuous counterpart of geometric. Constant hazard rate λ.
• Gamma(k, λ): sum of k independent Exponential(λ); mean k/λ, variance k/λ². Models total service time; conjugate prior for a Poisson rate.
• Normal(μ, σ²): the bell curve; symmetric, mean = median = mode. Standardise Z = (X−μ)/σ, then use the standard normal table. 68-95-99.7 within 1σ, 2σ, 3σ.
• Log-normal: X with ln X normal - right-skewed, positive only; income, salary, claim size, file size, web session time.
• Beta(α,β) on (0,1): models a probability or proportion; mean α/(α+β); conjugate prior for binomial p.
• Chi-square(ν) = sum of ν squared standard normals; t(ν) - heavier tails than normal, approaches normal as ν → ∞; F(ν₁,ν₂) - ratio of scaled chi-squares. These three are *sampling* distributions and reappear wherever variances and model comparisons are involved.
• Weibull(k, λ): flexible hazard - k < 1 decreasing (infant mortality), k = 1 exponential, k > 1 wear-out. Standard in reliability and life testing.
• Pareto / power law: heavy tail; city sizes, wealth, word frequencies, network degrees.

Joint, marginal and conditional distributions

• Joint PMF/PDF f(x,y) gives probability over pairs; marginals are obtained by summing or integrating out the other variable: f_X(x) = ∫f(x,y)dy.
• Conditional: f(y|x) = f(x,y)/f_X(x). Independence holds iff f(x,y) = f_X(x)f_Y(y) for all x, y.
• Covariance Cov(X,Y) = E[XY] − E[X]E[Y]; correlation ρ = Cov/(σ_Xσ_Y) ∈ [−1,1]. Independence ⇒ ρ = 0, but ρ = 0 does not imply independence (except for the bivariate normal).
• Var(X+Y) = Var X + Var Y + 2Cov(X,Y); for independent variables the covariance term vanishes.
• Law of total expectation: E[X] = E[E[X|Y]]; law of total variance: Var(X) = E[Var(X|Y)] + Var(E[X|Y]) - the bias-variance style decomposition.
• Multivariate normal N(μ, Σ) is specified by a mean vector and covariance matrix; all marginals and conditionals are normal. It underlies Gaussian discriminant analysis, Kalman filters, PCA assumptions and Gaussian processes.
• Transformations: for Y = g(X) monotone, f_Y(y) = f_X(g⁻¹(y)) |d g⁻¹/dy| (Jacobian) - this is exactly the change-of-variables rule used in normalising flows.

Limit theorems and approximations

• Law of Large Numbers: the sample mean converges to μ as n → ∞ (weak = in probability, strong = almost surely). Justifies Monte Carlo estimation.
• Central Limit Theorem: for i.i.d. X with finite mean μ and variance σ², the sample mean is approximately N(μ, σ²/n) for large n (n ≥ 30 is the usual working rule); hence Z = (X̄ − μ)/(σ/√n). It does not require the population to be normal, but it does require finite variance.
• Normal approximation to binomial (np and n(1−p) both ≥ 5), with a continuity correction of ±0.5; normal approximation to Poisson for λ ≳ 10.
• Poisson approximation to binomial for large n, small p.
• Standard error σ/√n shrinks as 1/√n - quadrupling the sample halves the error.

How distributions drive AI and machine learning

• Maximum likelihood = choosing parameters that maximise the assumed distribution's likelihood. Gaussian noise ⇒ squared-error loss; Bernoulli ⇒ binary cross-entropy; Poisson ⇒ Poisson regression (count data); multinomial ⇒ softmax cross-entropy.
• Generalised linear models pick the distribution of the response (normal, binomial, Poisson, gamma) and a link function.
• Naive Bayes assumes a distribution per feature: Gaussian for continuous, multinomial/Bernoulli for text counts.
• Entropy H(X) = −Σ p log p measures uncertainty; KL divergence D(P‖Q) = Σ p log(p/q) measures how far a model distribution is from the data distribution - minimising KL is equivalent to maximising likelihood. KL ≥ 0 and is asymmetric.
• Sampling: inverse transform, Box-Muller for normals, rejection sampling, and the reparameterisation trick (z = μ + σε, ε ~ N(0,1)) in variational autoencoders.
• Dropout is Bernoulli masking; weight initialisation (Xavier/He) sets variances so activations neither vanish nor explode; diffusion models add and remove Gaussian noise in steps.
• Indian practice note: no statute prescribes a distribution, but regulated modelling does - IRDAI mortality and claim-reserving work uses the Indian Assured Lives Mortality tables published by the Institute of Actuaries of India; RBI's expected-credit-loss and capital guidance requires loss-distribution-based provisioning; BIS/ISI quality-control standards use sampling plans built on hypergeometric and binomial/Poisson acceptance probabilities.

Key points

• PMF vs PDF - PMF gives actual probabilities; a PDF gives density, and only an area under it is a probability.
• CDF - F(x) = P(X ≤ x), defined for every random variable; differentiating it gives the PDF.
• Bernoulli/Binomial - binomial needs fixed n, independence and constant p; mean np, variance np(1−p).
• Hypergeometric vs binomial - without vs with replacement; they agree when the lot is large relative to the sample.
• Poisson - mean = variance = λ; if sample variance ≫ mean, the data are over-dispersed and negative binomial fits better.
• Geometric and exponential - the only memoryless distributions (discrete and continuous respectively).
• Normal - two parameters μ and σ²; standardise with Z = (X−μ)/σ; 68-95-99.7 rule.
• Log-normal - use it, not the normal, for strictly positive skewed quantities such as salary or claim size.
• Beta - distribution over a probability in (0,1); conjugate prior for binomial.
• Chi-square, t, F - sampling distributions derived from normals, used for variance, small-sample mean and ratio problems.
• Weibull - reliability workhorse; shape k decides whether failures are early-life, random or wear-out.
• Expectation is linear always; variance is not - variance adds only when covariance is zero.
• Zero correlation ≠ independence, except in the bivariate normal case.
• CLT - the sample mean tends to normal with variance σ²/n regardless of the parent distribution, provided variance is finite.
• Chebyshev - at least 1 − 1/k² of any distribution lies within k standard deviations, no shape assumed.
• MGF - M(t) = E[e^{tX}]; derivatives at 0 give moments, products give sums of independent variables.
• Loss ↔ distribution - MSE assumes Gaussian noise, cross-entropy assumes Bernoulli/multinomial.
• KL divergence - measures model-vs-data distribution gap; non-negative and asymmetric.

Formulas and examples

Symbols: n = number of trials, p = success probability, λ = rate (events per unit interval for Poisson, per unit time for exponential, so units are 1/time), k = count or shape, μ = mean (same units as X), σ = standard deviation (units of X), σ² = variance (units of X squared), ρ = dimensionless correlation, C(n,k) = n!/(k!(n−k)!).

Example 1 - Binomial, a campus placement test. Each of 10 multiple-choice questions has 4 options; a student guesses all of them. p = 0.25, n = 10.

• P(exactly 3 correct) = C(10,3)(0.25)³(0.75)⁷ = 120 × 0.015625 × 0.1334839 = 0.2503.
• P(0 correct) = 0.75¹⁰ = 0.0563; P(1) = 10 × 0.25 × 0.75⁹ = 10 × 0.25 × 0.0750847 = 0.1877; P(2) = 45 × 0.0625 × 0.75⁸ = 45 × 0.0625 × 0.1001129 = 0.2816.
• P(at least 3) = 1 − (0.0563 + 0.1877 + 0.2816) = 0.4744.
• Mean = np = 2.5, variance = 10 × 0.25 × 0.75 = 1.875, σ = 1.369.

Example 2 - Poisson, a support desk. A Bengaluru helpdesk receives on average 3 calls per 10-minute slot; calls are independent. λ = 3.

• P(no call) = e^{−3} = 0.0498.
• P(exactly 2) = e^{−3} × 3²/2! = 0.049787 × 4.5 = 0.2240.
• P(more than 3) = 1 − [P(0)+P(1)+P(2)+P(3)] = 1 − [0.049787 + 0.149361 + 0.224042 + 0.224042] = 1 − 0.647232 = 0.3528.
• Over a 30-minute window λ becomes 9, so mean = variance = 9 and σ = 3; P(no call in 30 min) = e^{−9} ≈ 0.000123.

Example 3 - Normal with standardisation, plus the empirical rule. Scores in an aptitude test are N(μ = 62, σ = 12).

• P(X > 80): Z = (80 − 62)/12 = 1.5 → P(Z > 1.5) = 1 − 0.9332 = 0.0668, about 6.7%.
• P(50 < X < 74): Z = −1 to +1 → 0.6826 (the 68% rule).
• The cut-off for the top 10%: Z = 1.2816 → x = 62 + 1.2816 × 12 = 77.4.
• Interquartile range: Z = ±0.6745 → 62 ± 8.09, i.e. 53.9 to 70.1.

Example 4 - Exponential, memorylessness and a machine. Time between breakdowns of a CNC machine is exponential with mean 200 hours, so λ = 1/200 = 0.005 per hour.

• P(runs more than 300 h) = e^{−0.005×300} = e^{−1.5} = 0.2231.
• P(fails within 100 h) = 1 − e^{−0.5} = 0.3935.
• Given it has already run 500 h, P(it runs 300 h more) = e^{−1.5} = 0.2231 again - memorylessness; past service tells nothing about remaining life.
• Variance = 1/λ² = 40,000 h², σ = 200 h (mean = σ for the exponential).

Example 5 - CLT and normal approximation to binomial. A food-delivery app's late-delivery rate is p = 0.20. In 400 orders, X = number late; np = 80, n(1−p) = 320, both ≥ 5.

• Mean 80, variance 400 × 0.2 × 0.8 = 64, σ = 8.
• P(X ≤ 70) with continuity correction: Z = (70.5 − 80)/8 = −1.1875 → 0.1175, about 11.8%.
• P(70 ≤ X ≤ 90): Z from (69.5−80)/8 = −1.3125 to (90.5−80)/8 = 1.3125 → 2(0.9053) − 1 = 0.8106.
• The sample proportion p̂ has SE = √(0.2×0.8/400) = √0.0004 = 0.02, so the late rate is estimated to within about ±0.039 (1.96 SE) at 95%.

import numpy as np
from scipy import stats

# Fit and compare: is the count data Poisson or over-dispersed?
x = np.random.poisson(3.0, size=5000)
print(x.mean(), x.var())            # both near 3 -> Poisson is plausible

# Tail probability of a normal, and a binomial exact value
print(1 - stats.norm.cdf(80, loc=62, scale=12))     # 0.0668
print(stats.binom.pmf(3, n=10, p=0.25))             # 0.2503

# Inverse-transform sampling from Exponential(lambda=0.005)
u = np.random.rand(10000)
t = -np.log(1 - u) / 0.005          # mean should be about 200 hours

Where it is used

• Banking, NBFCs and insurance: probability of default and loss-given-default distributions for RBI expected-credit-loss provisioning; claim frequency modelled as Poisson/negative binomial and claim severity as log-normal or gamma for IRDAI-regulated pricing and reserving; actuarial roles at LIC, HDFC Life, ICICI Lombard.
• E-commerce and delivery platforms (Flipkart, Swiggy, Zomato, Zepto): Poisson arrival of orders, exponential service times for rider-allocation and queueing models, log-normal delivery times, binomial conversion in A/B tests.
• Telecom and IT operations (Jio, Airtel, TCS, Infosys): call and packet arrivals as Poisson processes, Erlang/exponential service for capacity planning, Weibull for hardware reliability and spares forecasting.
• Manufacturing and quality (Tata Steel, Maruti, Bosch, pharma): binomial/hypergeometric acceptance sampling and Poisson defect counts in BIS-aligned sampling plans; normal-based process capability indices Cp and Cpk on Six Sigma projects.
• Data science and ML roles: choosing likelihoods and losses, noise models, priors, Monte Carlo simulation, Gaussian processes, VAEs and diffusion models; written tests at Fractal, Mu Sigma, ZS Associates, Quantiphi and American Express India lean heavily on this.
• Public data and weather/agri analytics: rainfall fitted with gamma distributions by IMD-style analyses, crop-yield and demand forecasting; NSSO/PLFS income data treated as log-normal.

Interview questions

• Q: What is the difference between a PMF and a PDF? A: A PMF gives the actual probability of each discrete value; a PDF gives density for a continuous variable, where only the area over an interval is a probability and P(X = x) = 0.
• Q: When would you use Poisson instead of binomial? A: When events are rare and the number of trials is large or undefined - you observe counts per unit of time or space rather than a fixed n, and np stays moderate while p → 0.
• Q: Your count data has variance far above the mean. What now? A: It is over-dispersed, so Poisson is misspecified; use a negative binomial (or quasi-Poisson) model, or check for unmodelled heterogeneity and zero inflation.
• Q: What does memorylessness mean and which distributions have it? A: The remaining waiting time does not depend on how long you have waited; the exponential (continuous) and geometric (discrete) are the only ones with this property.
• Q: State the Central Limit Theorem and one condition people forget. A: The sample mean of i.i.d. variables tends to N(μ, σ²/n) as n grows; the forgotten condition is finite variance - it fails for heavy-tailed laws like Cauchy or a Pareto with infinite variance.
• Q: Does zero correlation imply independence? A: No - correlation only captures linear association; Y = X² with X symmetric about 0 has ρ = 0 but is fully dependent. The exception is the bivariate normal, where ρ = 0 does imply independence.
• Q: Why do we use log-normal for salaries rather than normal? A: Salaries are strictly positive and right-skewed, arising from multiplicative effects; taking logs makes them roughly normal, and a normal model would assign probability to negative salaries.
• Q: How is the choice of loss function related to a distribution? A: Minimising a loss is maximising a likelihood - squared error corresponds to Gaussian noise, binary cross-entropy to Bernoulli, Poisson deviance to count data.
• Q: How would you sample from an arbitrary continuous distribution given only uniform random numbers? A: Inverse-transform sampling: draw U ~ Uniform(0,1) and return F⁻¹(U); if F⁻¹ is intractable, use rejection sampling or MCMC.
• Q: When would you prefer Chebyshev's inequality over a normal calculation? A: When the distribution's shape is unknown - Chebyshev needs only the mean and variance, at the cost of being a loose bound.
• Q: Why is the Weibull distribution preferred over the exponential in reliability? A: The exponential forces a constant hazard rate; Weibull's shape parameter can represent decreasing (early-life), constant or increasing (wear-out) hazard, matching real failure data.
• Q: What does KL divergence measure, and why is it not a distance? A: It measures how much information is lost using Q in place of P; it is non-negative but asymmetric and violates the triangle inequality, so it is a divergence, not a metric.

Common mistakes

• Treating a PDF value as a probability, or claiming P(X = x) > 0 for a continuous variable.
• Using binomial for sampling without replacement from a small lot, where hypergeometric is correct.
• Forgetting the continuity correction (±0.5) in the normal approximation to a binomial, or applying the approximation when np < 5.
• Mixing up λ as a rate and 1/λ as a mean in the exponential, and failing to rescale λ when the interval length changes (3 per 10 min is 9 per 30 min).
• Adding variances of dependent variables without the 2Cov(X,Y) term, or assuming E[g(X)] = g(E[X]) (true only for linear g).
• Assuming the CLT makes the *data* normal - it applies to the sampling distribution of the mean, not to the individual observations.
• Reading standard normal tables wrongly - mixing a one-sided table with a "area between ±z" table, and forgetting symmetry for negative z.

Related topics

Descriptive statistics · Statistical inference and estimation · Hypothesis testing · Bayesian statistics · Experiment design and A/B testing · Causal inference basics