Sampling methods and survey design

In short

Sampling is the practice of selecting a subset of units from a population so that valid conclusions about the whole can be drawn from the part, and survey design is the surrounding process of defining the frame, instrument and data collection. It matters because every real dataset in analytics — a customer survey, an A/B cohort, a credit audit, an NSS household round — is a sample, and bias introduced here cannot be fixed later by modelling. It is examined in B.Sc Data Science statistics papers and asked in analytics interviews (market research firms, Nielsen, Kantar, banks, product analytics teams) as "how would you sample this?"

Explained

Basic vocabulary and why we sample

• Population: all units of interest (e.g. all 1.4 crore households in a state). Sample: the subset actually observed. Sampling unit: the thing selected (household, person, invoice). Sampling frame: the list from which units are drawn — electoral rolls, census blocks, a CRM table.
• Reasons to sample: cost, time, feasibility, and sometimes greater accuracy (a small well-supervised survey beats a huge sloppy census).
• Parameter vs statistic: population mean μ, proportion P are parameters; sample x̄, p are statistics used as estimators.
• Sampling error: the natural difference between statistic and parameter, measurable and reducible by increasing n. Non-sampling error: frame gaps, non-response, measurement and recall error, data entry mistakes — not reduced by a bigger sample, often the larger error in practice.

Probability sampling methods

1. Simple random sampling (SRS): every unit has equal chance; drawn with replacement (WR) or without replacement (WOR) using random numbers or a lottery. Basis of all theory, but needs a complete frame and may miss small groups.
2. Stratified random sampling: divide the population into homogeneous, non-overlapping strata (state, income class, product category), then sample independently in each. Gives lower variance than SRS when strata differ; also allows stratum-level estimates. Allocation can be equal, proportional (nᵢ ∝ Nᵢ) or Neyman/optimum (nᵢ ∝ NᵢSᵢ).
3. Systematic sampling: pick a random start r in 1…k and then every k-th unit, k = N/n. Simple and fast on an ordered list, but biased if the list has a periodicity matching k.
4. Cluster sampling: divide into naturally occurring groups (villages, wards, colleges), randomly select whole clusters and survey all units inside. Cheap when travel dominates cost; less precise because units within a cluster are similar (high intra-class correlation).
5. Multistage and PPS: select districts, then villages, then households — the design used by India's National Sample Survey (NSS) and NFHS. PPS (probability proportional to size) gives larger units a higher selection chance, with weights 1/πᵢ used in estimation.
6. Non-probability methods: convenience, judgement/purposive, quota, snowball (for hidden populations). Quick and cheap, but selection probabilities are unknown, so standard errors and confidence intervals are not strictly valid — the flaw behind most "online poll" failures.

Survey design and data quality

• Steps: define objective and target population → choose frame and sampling design → fix sample size and precision → design the questionnaire → pilot/pre-test → train enumerators → collect (CAPI/CATI/web/face-to-face) → edit, impute, weight → estimate with standard errors → report limitations.
• Questionnaire rules: short simple wording, one idea per question, no double-barrelled or leading questions, balanced response scales, sensitive items late, local-language translation and back-translation (vital in Indian multilingual surveys), skip logic and consistency checks.
• Question types: open-ended, closed (dichotomous, multiple choice), Likert scales (usually 5 or 7 point), ranking.
• Bias sources: selection bias (frame excludes a group — e.g. a web survey missing households without internet access), non-response bias (those who answer differ from those who do not), response/social-desirability bias, interviewer bias, recall bias, survivorship bias.
• Remedies: call-backs and reminders, incentives, weighting adjustment and post-stratification/raking to known Census totals, follow-up subsample of non-respondents, anonymity assurances.
• Indian practice and law: NSS/Periodic Labour Force Survey rounds use stratified multistage designs with Census blocks/UFS frames; official statistics follow the MoSPI framework. Personal data collected in surveys is governed by the Digital Personal Data Protection Act, 2023 — collect only with notice and consent, limit purpose, allow withdrawal (rules were being notified in phases, so check current status).

Key points

• Sampling frame - the operational list used for selection; gaps in it cause coverage bias no sample size can cure.
• Probability sampling - every unit has a known, non-zero chance of selection; only then are SEs and CIs valid.
• SRS - equal probability; variance of mean = σ²/n (WR) or (σ²/n)(N−n)/(N−1) (WOR).
• Stratification - reduces variance by making within-group variation small and between-group variation large.
• Cluster sampling - cheapest per unit but least precise; precision loss measured by the design effect (deff).
• Systematic sampling - random start plus fixed interval k = N/n; avoid when data are cyclic.
• PPS/multistage - the real-world design for large national surveys; needs sampling weights in estimation.
• Standard error - falls as 1/√n, so quadrupling the sample halves the error; diminishing returns.
• Finite population correction - (N−n)/N matters only when n/N exceeds about 5%.
• Non-sampling error - non-response, recall and measurement errors, usually the dominant error in field surveys.
• Quota vs stratified - quota fills counts by convenience; stratified selects randomly within strata.
• Pilot survey - small pre-test to estimate variance, timing and question problems before full rollout.

Formulas and examples

Symbols: N population size, n sample size, σ population SD, Sᵢ stratum SD, E margin of error (absolute), z the normal value (1.96 at 95%), p sample proportion.

Example 1 — sample size for a proportion, with FPC. A bank wants the proportion of 50,000 credit-card customers who would take a loan top-up, to within ±4 percentage points at 95% confidence. Worst-case p = 0.5.

• n₀ = (1.96² × 0.5 × 0.5)/0.04² = (3.8416 × 0.25)/0.0016 = 0.9604/0.0016 = 600.25 → 601.
• FPC: n = 600.25 / (1 + 600.25/50,000) = 600.25/1.012005 = 593.1 → 594 customers.
• Suppose 594 are surveyed and 198 say yes: p = 198/594 = 0.3333; SE = √(0.3333 × 0.6667/594) = √0.000374 = 0.01934 → 1.93 pp.
• 95% CI = 0.3333 ± 1.96 × 0.01934 = 0.3333 ± 0.0379 → (29.5%, 37.1%).

Example 2 — proportional vs Neyman allocation. A state survey of monthly household spend has three strata: urban N₁ = 20,000 (S₁ = ₹4,000), semi-urban N₂ = 30,000 (S₂ = ₹2,000), rural N₃ = 50,000 (S₃ = ₹1,000). Total N = 1,00,000, sample n = 1,000.

• ΣNᵢSᵢ = 8 + 6 + 5 = 19 crore. Neyman: urban = 1,000 × 8/19 = 421, semi-urban = 1,000 × 6/19 = 316, rural = 1,000 × 5/19 = 263 (these are the figures in the table; they sum to 1,000).
• Interpretation: Neyman shifts effort to the high-variability urban stratum, lowering the variance of the overall mean for the same cost.

Where it is used

• Market research and media measurement: Kantar, Nielsen, IPSOS, BARC panels — stratified multistage samples of households by state, town class and socio-economic group.
• Official statistics and policy: MoSPI's NSS and PLFS rounds, NFHS, RBI consumer confidence and inflation expectations surveys; results drive GDP, unemployment and poverty estimates.
• Business analytics roles: data analysts sampling transaction logs for quality audits, credit-card fraud sampling, telecom NPS/CSAT surveys, retail audits, and sampling plans for statutory and internal audits.
• Product and growth teams: sampling users for usability studies and survey panels, and weighting web-panel results back to the user base.

Interview questions

• Q: When is stratified sampling better than simple random sampling? A: When the population splits into groups that are internally homogeneous but differ from each other; stratifying then gives a smaller variance for the same n and also yields group-level estimates.
• Q: What is the difference between sampling error and non-sampling error? A: Sampling error is the chance variation from observing only part of the population and shrinks as n grows; non-sampling error comes from frame gaps, non-response and measurement, and does not shrink with n.
• Q: Why can't you compute a valid confidence interval from a convenience sample? A: Because selection probabilities are unknown, so the estimator's sampling distribution is undefined and any reported standard error rests on an assumption that is not satisfied.
• Q: What goes wrong with systematic sampling? A: If the ordered list has a cycle whose period matches the interval k, the sample systematically over- or under-represents a type of unit, biasing the estimate.
• Q: Why does a national survey use cluster/multistage sampling despite the precision loss? A: Travel and listing costs dominate; concentrating fieldwork in selected villages or blocks lets you survey far more units per rupee, and the larger n partly offsets the design effect.
• Q: How do you handle non-response? A: Make call-backs, then adjust with weighting or post-stratification to known population totals, and ideally survey a subsample of non-respondents to test whether they differ.
• Q: How do you decide sample size for a proportion? A: Fix the confidence level and margin of error, use n = z²p(1−p)/E² with p = 0.5 if p is unknown, then apply the finite population correction if n exceeds about 5% of N.

Common mistakes

• Treating a large sample as a good sample — a biased frame (online-only, walk-in customers) stays biased at any n.
• Confusing quota sampling with stratified sampling, or calling a purposive sample "random".
• Using n₀ without the finite population correction when the sample is a sizeable share of a small population.
• Ignoring sampling weights and the design effect when analysing multistage/PPS survey data, which understates standard errors.
• Mixing up population SD σ with sample SD s, or using a margin of error in percent while p is in proportion.

Related topics

Descriptive statistics · Statistical inference and estimation · Hypothesis testing · Experiment design and A/B testing · Correlation and regression