Descriptive statistics

In short

Descriptive statistics is the set of methods that summarise a data set — its centre, spread, shape and the association between variables — without drawing conclusions about a larger population. It is the first step of every machine-learning project: exploratory data analysis (EDA), feature scaling, outlier handling and data-quality checks all rest on these numbers. In exams it appears in Mathematics/Statistics for AI papers and in the GATE Data Science & AI (DA) paper, which lists central tendency, dispersion, correlation and covariance explicitly. In Indian hiring, analyst and data-scientist rounds at TCS, Infosys, Wipro, Accenture, Mu Sigma, Fractal, ZS Associates, Flipkart, Swiggy and bank analytics teams routinely begin with mean-versus-median, standard deviation and outlier questions, followed by a pandas or SQL implementation.

Explained

Data, variables and scales of measurement

• Population vs sample: population = every unit of interest (size N); sample = the observed subset (size n). Descriptive statistics describe whatever data you hold; inference generalises — that is a different topic.
• Parameter vs statistic: a parameter (μ, σ², ρ) describes a population; a statistic (x̄, s², r) is computed from a sample and estimates it.
• Variable types: qualitative/categorical (nominal — blood group, state; ordinal — grade, Likert scale) and quantitative (discrete — number of defects; continuous — height, salary).
• Scales: nominal (labels only), ordinal (order, no equal gaps), interval (equal gaps, arbitrary zero — temperature in °C), ratio (true zero — income in ₹, mass in kg). The scale fixes which statistic is legal: mode for nominal, median for ordinal, mean onwards for interval/ratio.
• Cross-sectional, time-series and panel data; univariate, bivariate, multivariate analysis.
• Frequency distribution: raw data → classes with class limits, class boundaries, class mark (mid-value), class width, frequency f, relative frequency f/n, cumulative frequency (less-than and more-than ogives).
• Graphical summaries: bar chart and pie chart (categorical), histogram (continuous; area ∝ frequency, so use frequency density when class widths differ), frequency polygon, ogive, stem-and-leaf, boxplot, violin plot, scatter plot, Q-Q plot, heatmap of a correlation matrix, Pareto chart.

Measures of central tendency

• Arithmetic mean (AM): sum ÷ count. Uses every value, is algebraically tractable (the only measure with Σ(xᵢ − x̄) = 0), but is pulled by outliers and undefined for open-ended classes.
• Weighted mean: used for CGPA (credits as weights), index numbers and class-imbalanced metrics.
• Median: the middle value of ordered data; splits the data 50-50. Robust to outliers and usable with open-ended classes; ignores the magnitudes of extreme values.
• Mode: most frequent value; the only measure valid for nominal data (modal shirt size, modal complaint type). May not exist or may be multiple (bimodal, multimodal).
• Geometric mean (GM): nth root of the product — correct for growth rates, returns, ratios and index numbers (CAGR). Needs all positive values.
• Harmonic mean (HM): reciprocal of the mean of reciprocals — correct for averaging rates over equal distances (speed) and for the F1-score (HM of precision and recall).
• Relation: for positive, non-identical values AM > GM > HM; AM = GM = HM only when all values are equal. Also AM × HM = GM² for exactly two observations.
• Trimmed mean / Winsorised mean: drop or cap the extreme α% at each end — a middle ground between mean and median, used in robust pipelines.
• Empirical relation (moderately skewed data): Mode ≈ 3 Median − 2 Mean.
• Positional measures: quartiles Q₁, Q₂, Q₃; deciles D₁…D₉; percentiles P₁…P₉₉. P₅₀ = Q₂ = median. Percentile rank is how CAT/JEE scores are reported.

Measures of dispersion

• Range = max − min: simplest, uses only two values, very unstable. Coefficient of range = (max − min)/(max + min).
• Quartile deviation (semi-interquartile range) = (Q₃ − Q₁)/2; IQR = Q₃ − Q₁ — robust, the basis of boxplots.
• Mean absolute deviation (MAD) = average |xᵢ − A|; smallest when A = median. Not differentiable at zero, hence less used in optimisation.
• Variance and standard deviation: average squared deviation from the mean. SD carries the same unit as the data (₹, kg, marks) and is the workhorse of statistics because it is differentiable and additive for independent variables.
• Bessel's correction: divide by (n − 1) for the sample variance so that E[s²] = σ². Dividing by n gives a biased (under-estimating) variance. numpy.var defaults to ddof=0 (population); pandas.Series.var and df.describe() default to ddof=1 (sample) — a classic interview catch.
• Standard error of the mean = s/√n: spread of the *sample mean*, not of the data; falls as 1/√n.
• Coefficient of variation (CV) = SD/mean × 100%: unit-free, used to compare variability of series with different units or very different means (rainfall across districts, returns of two stocks). Meaningless when the mean is near zero or negative.
• Properties of variance: Var(X + c) = Var(X); Var(aX) = a²Var(X); for independent X, Y, Var(X ± Y) = Var(X) + Var(Y). Shift changes the mean but not the SD.
• Combined (pooled) mean and variance let two groups be merged without the raw data — a standard exam question and the logic behind streaming/online statistics.
• Chebyshev's inequality: for *any* distribution, at least (1 − 1/k²) of data lies within k SDs of the mean (≥75% for k = 2, ≥88.9% for k = 3).
• Empirical (68-95-99.7) rule: for an approximately normal distribution, about 68%, 95% and 99.7% of data lie within 1, 2 and 3 SDs.

Moments, skewness and kurtosis

• Raw moments about origin: μ'ᵣ = Σxᵢʳ/n. Central moments about the mean: μᵣ = Σ(xᵢ − x̄)ʳ/n, so μ₁ = 0 and μ₂ = variance.
• Conversions: μ₂ = μ'₂ − μ'₁²; μ₃ = μ'₃ − 3μ'₂μ'₁ + 2μ'₁³.
• Skewness = asymmetry. Positive (right) skew: long right tail, Mean > Median > Mode — Indian income, salary, insurance claim and e-commerce order-value data. Negative (left) skew: Mean < Median < Mode.
• Measures: Karl Pearson's coefficient Sk = (Mean − Mode)/SD (or 3(Mean − Median)/SD); Bowley's quartile coefficient (bounded −1 to +1); moment coefficient γ₁ = μ₃/μ₂^1.5 (Pearson's β₁ = μ₃²/μ₂³ = γ₁², which carries no sign). Rule of thumb: |skew| < 0.5 ≈ symmetric, > 1 strongly skewed.
• Kurtosis β₂ = μ₄/μ₂² measures tail weight/peakedness. Normal has β₂ = 3; excess kurtosis γ₂ = β₂ − 3. Leptokurtic (γ₂ > 0, fat tails — stock returns), platykurtic (γ₂ < 0), mesokurtic (= 0).
• Why it matters in ML: heavy skew breaks linear models and distance metrics, so log, square-root or Box-Cox transforms and quantile binning are applied; fat tails mean more outliers than a normal assumption predicts.

Bivariate description: covariance and correlation

• Covariance Cov(X, Y) = E[(X − μx)(Y − μy)]: sign gives the direction of linear association, but its magnitude depends on units (₹·kg), so it cannot be compared across pairs.
• Pearson's correlation coefficient r = Cov(X, Y)/(sx·sy): unit-free, −1 ≤ r ≤ +1. r = ±1 means perfect linear relation; r = 0 means no *linear* relation (y = x² on [−1, 1] gives r = 0 with perfect dependence).
• Properties: r is unchanged by a change of origin or by multiplication by positive constants (a negative multiplier flips the sign); r² (coefficient of determination) is the fraction of variance explained by the linear fit; r = ±√(b_yx · b_xy) where b are the regression slopes.
• Spearman's rank correlation ρ: Pearson's r on ranks — captures any monotonic relation, resistant to outliers, the right choice for ordinal data. Kendall's tau uses concordant/discordant pairs.
• Correlation matrix / heatmap is used in EDA to spot multicollinearity (|r| > 0.8 between features) before regression.
• Categorical association: contingency (two-way) tables with row/column/overall percentages, mode per cell; Cramér's V. Watch for Simpson's paradox — an association that reverses after grouping.
• Correlation is not causation; spurious correlation arises from a lurking variable. (Causal identification belongs to the causal-inference topic.)

Grouped data, outliers and practice

• Grouped (frequency-table) formulas treat every observation as sitting at its class mark, so results are approximations; the step-deviation method keeps the arithmetic small.
• Median and mode of grouped data are interpolated inside the median/modal class using the standard formulas given below.
• Outlier rules: (a) IQR/Tukey rule — below Q₁ − 1.5·IQR or above Q₃ + 1.5·IQR is mild, 3·IQR is extreme; (b) z-score |z| > 3; (c) robust z using median and MAD (modified z = 0.6745(x − median)/MAD, flag > 3.5). The IQR and MAD rules are preferred because the mean and SD are themselves distorted by outliers (masking).
• Boxplot (five-number summary): minimum, Q₁, median, Q₃, maximum, with whiskers at the Tukey fences and points beyond plotted individually.
• Missing data in the summary stage: report the count of non-nulls alongside each statistic; mean/median imputation changes the SD (shrinks it), so always state it.
• Standardisation vs normalisation: z = (x − x̄)/s gives mean 0, SD 1 (used by SVM, PCA, k-means, neural nets); min-max scaling maps to [0, 1] but is outlier-sensitive; robust scaling uses median and IQR. Statistics computed on the training set only must be reused on validation and test data to avoid leakage.
• Indian official practice: NSSO/NSO sample surveys (PLFS, HCES), the Census and RBI/MoSPI releases publish weighted means, medians and percentile distributions; survey weights (multipliers) must be used, so the plain mean of a PLFS microdata column is wrong. The Collection of Statistics Act, 2008 (amended 2017) governs official statistical collection in India, and MoSPI maintains the National Quality Assurance Framework for official statistics. For industrial quality work, control charts (X̄-R, X̄-s) built on means and SDs follow IS 397 / ISO 7870 series — the exact current part numbers are revised periodically, so verify before citing.

Key points

• Descriptive vs inferential - describing the data in hand vs generalising to a population.
• Mean - uses all values, minimises Σ(x − a)², sensitive to outliers; Σ(xᵢ − x̄) = 0 always.
• Median - the 50th percentile, robust, minimises Σ|x − a|; preferred for skewed data such as income.
• Mode - highest-frequency value, the only valid centre for nominal data, may be non-unique.
• GM - for growth rates and CAGR; HM - for rates and the F1-score; AM ≥ GM ≥ HM.
• Range and IQR - range uses two extremes only; IQR = Q₃ − Q₁ is robust and drives boxplots.
• Variance/SD - average squared deviation; SD shares the data's unit; variance is additive for independent variables.
• Bessel's correction - divide by n − 1 for an unbiased sample variance; numpy's default ddof=0 differs from pandas'.
• CV - SD/mean as a %, the right way to compare spread across different units or scales.
• Chebyshev - ≥ 1 − 1/k² of any data within k SDs; the 68-95-99.7 rule needs normality.
• Skewness - direction of the tail; right-skewed data have Mean > Median > Mode.
• Kurtosis - tail weight; normal β₂ = 3, excess kurtosis = β₂ − 3; leptokurtic = fat tails.
• Moments - μ₁ = 0, μ₂ = variance, μ₃ → skewness, μ₄ → kurtosis.
• Covariance - direction of linear association, unit-dependent; correlation - standardised to [−1, 1].
• r = 0 ≠ independence - it only rules out a *linear* relation; use Spearman for monotonic patterns.
• r² - the share of variance explained; r is invariant to shift and positive scaling.
• Tukey fences - Q₁ − 1.5·IQR and Q₃ + 1.5·IQR mark outliers; z > 3 or modified z > 3.5 are alternatives.
• Scaling leakage - compute mean/SD on train data only, then apply the same values to test data.

Formulas and examples

Example 1 — Mean, median, mode, SD, CV and skew on a salary sample (₹ lakh per annum). Campus offers: 4, 5, 5, 6, 7, 8, 45 (n = 7).

• Σx = 4+5+5+6+7+8+45 = 80 → x̄ = 80/7 = ₹11.43 LPA.
• Ordered data, 4th value = median ₹6 LPA; mode = ₹5 LPA (appears twice).
• Σx² = 16+25+25+36+49+64+2025 = 2240. s² = [2240 − 80²/7]/6 = [2240 − 914.286]/6 = 1325.714/6 = 220.95 → s = ₹14.86 LPA.
• CV = 14.86/11.43 × 100 = 130% — enormous spread relative to the centre.
• Pearson skew = 3(11.43 − 6)/14.86 = +1.10 → strongly right-skewed. Mean > Median > Mode, confirming the direction.
• Quartiles (exclusive/Tukey style with the median split): lower half 4, 5, 5 → Q₁ = 5; upper half 7, 8, 45 → Q₃ = 8. IQR = 3; upper fence = 8 + 4.5 = 12.5 → 45 is an outlier.
• Reporting rule: the median ₹6 LPA describes a typical offer; the mean ₹11.43 LPA is distorted by one package. This is exactly why placement brochures quoting "average package" are misleading.

Example 2 — Grouped data: marks of 50 students.

• Mean = 1410/50 = 28.2 marks.
• n/2 = 25 → median class 20-30 (CF before = 12, f = 14, L = 20, h = 10). Median = 20 + [(25 − 12)/14]×10 = 20 + 9.286 = 29.29 marks.
• Modal class 30-40: f₀ = 14, f₁ = 16, f₂ = 8. Mode = 30 + [(16 − 14)/(32 − 14 − 8)]×10 = 30 + (2/10)×10 = 32 marks.
• Check with the empirical relation: 3(29.29) − 2(28.2) = 87.87 − 56.4 = 31.5 ≈ 32 — consistent.
• Q₁: n/4 = 12.5 → class 20-30 → 20 + [(12.5 − 12)/14]×10 = 20.36. Q₃: 3n/4 = 37.5 → class 30-40 → 30 + [(37.5 − 26)/16]×10 = 37.19. IQR = 16.83, QD = 8.41.
• Bowley skew = (37.19 + 20.36 − 2×29.29)/16.83 = (57.55 − 58.58)/16.83 = −0.061 → almost symmetric, slight left skew.

Example 3 — Covariance, Pearson r and r². Hours studied (x) vs test score (y) for 5 students: (2, 50), (4, 60), (6, 70), (8, 75), (10, 90).

• n = 5, Σx = 30, Σy = 345, Σxy = 100+240+420+600+900 = 2260, Σx² = 4+16+36+64+100 = 220, Σy² = 2500+3600+4900+5625+8100 = 24725.
• x̄ = 6, ȳ = 69.
• Numerator: nΣxy − ΣxΣy = 5(2260) − 30(345) = 11300 − 10350 = 950.
• nΣx² − (Σx)² = 1100 − 900 = 200; nΣy² − (Σy)² = 123625 − 119025 = 4600.
• r = 950/√(200 × 4600) = 950/√920000 = 950/959.17 = 0.990 → very strong positive linear relation; r² = 0.981, so 98.1% of score variance is explained linearly by hours.
• Cov (sample) = [Σxy − nx̄ȳ]/(n − 1) = [2260 − 5×6×69]/4 = (2260 − 2070)/4 = 47.5 (unit: hour·marks).
• Sanity check: s_x = √(200/(5×4)) = √10 = 3.162; s_y = √(4600/20) = √230 = 15.166; r = 47.5/(3.162 × 15.166) = 47.5/47.96 = 0.990 ✓.

Example 4 — Combined mean and variance of two batches. Batch A: n₁ = 30, x̄₁ = 60, σ₁ = 5. Batch B: n₂ = 20, x̄₂ = 70, σ₂ = 8.

• Combined mean = (30×60 + 20×70)/50 = (1800 + 1400)/50 = 64.
• d₁ = 60 − 64 = −4; d₂ = 70 − 64 = +6.
• σ² = [30(25 + 16) + 20(64 + 36)]/50 = [30(41) + 20(100)]/50 = (1230 + 2000)/50 = 64.6 → σ = 8.04.
• Note the combined SD (8.04) exceeds both group SDs because the difference in group means adds between-group variability.

Example 5 — z-scores and comparing across different scales. A candidate scores 78 in Quant (mean 60, SD 12) and 85 in Verbal (mean 75, SD 20).

• z_Quant = (78 − 60)/12 = +1.50; z_Verbal = (85 − 75)/20 = +0.50.
• Despite the lower raw mark, the Quant performance is relatively far stronger (1.5 SD above the mean vs 0.5 SD).
• Under a normal assumption, +1.50 ≈ 93rd percentile and +0.50 ≈ 69th percentile. Chebyshev alone (no normality) would only guarantee that at least 1 − 1/1.5² = 55.6% of candidates lie within ±1.5 SD.

import pandas as pd
df = pd.read_csv("offers.csv")
df.describe()                       # count, mean, std (ddof=1), min, 25%, 50%, 75%, max
df["ctc"].agg(["mean","median","std","skew","kurt"])   # kurt() returns EXCESS kurtosis
q1, q3 = df["ctc"].quantile([0.25, 0.75])
iqr = q3 - q1
mask = (df["ctc"] < q1 - 1.5*iqr) | (df["ctc"] > q3 + 1.5*iqr)   # Tukey outliers
df.corr(numeric_only=True)          # Pearson; method="spearman" for ranks

Where it is used

• Analytics and data-science roles (Mu Sigma, Fractal, ZS, Quantiphi, TCS/Infosys/Wipro analytics): EDA notebooks, data-quality reports and feature engineering begin with describe(), distributions, correlation heatmaps and outlier treatment.
• Banking, NBFC and insurance (HDFC, ICICI, Bajaj Finserv, LIC): credit-score and claim-amount distributions, percentile-based risk cut-offs, variance/CV-based portfolio and premium pricing inputs.
• Manufacturing and quality (Tata Steel, Maruti, Bosch, pharma plants): SPC control charts and process capability indices built from the mean and SD; Six Sigma Define-Measure phases.
• Product and operations (Flipkart, Swiggy, Zomato, Ola, PhonePe): median and p95/p99 latency and delivery-time dashboards, AOV distributions, funnel drop-off summaries — tail percentiles, not averages, drive SLAs.
• Public policy and official statistics: MoSPI/NSO surveys (PLFS, HCES), CPI/WPI index numbers (geometric means of price relatives), RBI bulletins, NITI Aayog indices — all reported as weighted means and percentile distributions.
• Healthcare and public health (ICMR, hospital analytics): summary tables of patient cohorts (mean ± SD or median with IQR), reference ranges set as central percentiles.

Interview questions

• Q: When do you prefer the median over the mean? A: When the data are skewed or contain outliers, or the scale is only ordinal — e.g. Indian salary or house-price data, where a few very large values drag the mean away from a typical value.
• Q: Why divide by n − 1 in the sample variance? A: Deviations are taken from the sample mean, which costs one degree of freedom; dividing by n − 1 makes the estimator unbiased, so E[s²] = σ².
• Q: Standard deviation versus standard error? A: SD measures the spread of individual observations; SE = s/√n measures the spread of the sample mean and shrinks as the sample grows.
• Q: Covariance or correlation — which would you report, and why? A: Correlation, because it is unit-free and bounded in [−1, 1], so values are comparable across variable pairs; covariance's magnitude depends on the units.
• Q: Can r be zero when X and Y are strongly related? A: Yes — r only detects linear association; a U-shaped relation such as y = x² on a symmetric range gives r ≈ 0 despite perfect dependence.
• Q: How do you detect outliers without assuming normality? A: Use the Tukey IQR fences (Q₁ − 1.5·IQR, Q₃ + 1.5·IQR) or a modified z-score based on the median and MAD, since the mean and SD are themselves inflated by the outliers.
• Q: What does a CV of 130% tell you? A: The standard deviation exceeds the mean, so the series is extremely dispersed — typically a heavily right-skewed variable where the mean is not a useful summary.
• Q: Why is the geometric mean used for returns? A: Returns compound multiplicatively, so the GM of growth factors reproduces the actual final value (CAGR); the arithmetic mean overstates realised growth.
• Q: Which average gives the F1-score, and why? A: The harmonic mean of precision and recall — it penalises imbalance, so a model that is strong on one and weak on the other cannot score well.
• Q: Chebyshev's inequality versus the empirical rule? A: Chebyshev holds for any distribution but is loose (≥75% within 2 SD); the 68-95-99.7 rule is tight but requires approximate normality.
• Q: You standardise features before k-means. What exactly do you compute and where? A: Mean and SD on the training set only, then transform train, validation and test with those same values — recomputing on test data leaks information and shifts the scale.
• Q: What is Simpson's paradox? A: An association present in aggregated data reverses or disappears within subgroups because of a confounding variable, so always inspect summaries by relevant groups.

Common mistakes

• Quoting the mean for strongly skewed data (salaries, latencies, claim sizes) instead of the median with the IQR, and quoting "average package" without the distribution.
• Mixing up ddof: using numpy.var (population, ddof=0) where the sample variance is required, or vice versa, and getting a different answer from pandas.
• Forgetting that mean/variance of grouped data are approximations, or using class limits instead of class marks and class boundaries.
• Treating correlation as causation, or reading r = 0 as independence; also ignoring that a single outlier can create or destroy a high r.
• Applying the 68-95-99.7 rule to non-normal data, and using z > 3 to screen outliers in a heavy-tailed distribution where the SD is already inflated.
• Averaging rates or percentages with the arithmetic mean (speeds, growth rates, ratios) where the harmonic or geometric mean is required; averaging already-averaged values without weights.
• Computing scaling statistics on the full data set before the train-test split, and comparing CV for interval data with an arbitrary zero or a near-zero mean.

Related topics

Probability distributions · Statistical inference and estimation · Hypothesis testing · Experiment design and A/B testing · Linear algebra for machine learning · Causal inference basics