Systematic reviews and meta-analysis

In short

A systematic review answers one focused question by finding, appraising and summarising every eligible study using a pre-registered, reproducible protocol; meta-analysis is the optional statistical step that pools their effect estimates into one weighted summary with a confidence interval. It sits at the top of the evidence hierarchy because it removes the selection of convenient studies that makes a narrative review unreliable. In MPH programmes it is examined as a full unit (protocol writing, PRISMA flow, forest plot reading, heterogeneity statistics, bias assessment) and is a standard dissertation format. In India it is the engine behind ICMR and MoHFW guidelines, HTAIn technology assessments, NLEM revisions and Standard Treatment Guidelines, and it is what research-associate, HTA-analyst, epidemiologist and medical-writing interviews at ICMR institutes, PHFI/IIPHs, AIIMS, CMC Vellore (Cochrane South Asia), NHSRC, CROs and global-health NGOs test.

Explained

What it is, and the family of review types

• Systematic review: explicit question, pre-specified eligibility criteria, exhaustive search of multiple databases, duplicate screening and data extraction, formal risk-of-bias assessment, and a transparent synthesis. Narrative ("literature") reviews have none of these safeguards.
• Meta-analysis: quantitative pooling. A systematic review may legitimately end in a narrative/structured synthesis (SWiM reporting) when outcomes, populations or designs are too unlike to pool.
• Types: intervention review (RCTs), prevalence/proportion review (very common in Indian public health), prognostic review, diagnostic test accuracy review, aetiological review of observational studies, qualitative evidence synthesis (meta-ethnography, ENTREQ), scoping review (maps the field, PRISMA-ScR, no pooling), rapid review, umbrella/overview of reviews, living systematic review (continuously updated), individual participant data (IPD) meta-analysis, network meta-analysis (NMA), dose-response meta-analysis.
• Levels: meta-analysis of aggregate (published) data is cheap but limited to what authors report; IPD-MA allows uniform analysis, time-to-event re-analysis and genuine subgroup work, and is the gold standard.

The steps (the examinable sequence)

1. Frame the question — PICO (Population, Intervention, Comparator, Outcome; add T for time, S for study design); PECO for exposures; PIRD/PIT for diagnostic accuracy; SPIDER/SPICE for qualitative.
2. Write and register a protocol — PROSPERO (CRD, University of York) before screening starts; INPLASY and OSF are alternatives; Cochrane protocols are peer-reviewed and published. Registration is the main defence against outcome switching and selective reporting. Report the protocol using PRISMA-P.
3. Search — at least MEDLINE/PubMed, Embase and Cochrane CENTRAL; add Scopus, Web of Science, CINAHL, PsycINFO, Global Index Medicus/WHO IRIS, and for India IndMED/medIND (ICMR-NIC), CTRI (Clinical Trials Registry – India, run by ICMR-NIMS; prospective registration of trials in India is required), Shodhganga for theses, and ministry/NHSRC reports as grey literature. Use MeSH/Emtree plus free text, Boolean operators, truncation and validated filters; hand-search reference lists, do citation chaining, contact authors. Report the full strategy (PRISMA-S) with dates.
4. Screen — de-duplicate, then title/abstract and full-text screening by two independent reviewers, disagreements resolved by a third; measure agreement with Cohen's or Fleiss' kappa. Tools: Rayyan, Covidence, EPPI-Reviewer, Zotero/EndNote.
5. Extract data — piloted form, duplicate extraction: design, setting, sample size, participant characteristics, intervention detail, outcome definitions, effect estimates with variance, funding and conflicts.
6. Assess risk of bias (study level) — Cochrane RoB 2 for RCTs: randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, selection of the reported result → low / some concerns / high. ROBINS-I (7 domains, pre- and post-intervention) or Newcastle–Ottawa Scale for observational studies; QUADAS-2 for diagnostic accuracy; PROBAST for prediction models. (Quality scores are discouraged; domain-based judgements are used.)
7. Synthesise — choose the effect measure, pool with an appropriate model, quantify heterogeneity, explore it, test robustness.
8. Assess certainty (body of evidence) — GRADE: RCT bodies start High, observational start Low; downgrade for risk of bias, inconsistency, indirectness, imprecision, publication bias; upgrade observational evidence for large effect, dose–response or when plausible residual confounding would shrink the effect. Present a Summary of Findings table.
9. Report — PRISMA 2020 (27 items, 7 sections, plus abstract checklist) with the flow diagram; MOOSE for observational reviews, PRISMA-DTA, PRISMA-NMA, PRISMA-IPD extensions. The review itself is appraised with AMSTAR-2 (16 items, 7 critical domains) or ROBIS.

Choosing the effect measure

• Binary outcomes: risk ratio (RR), odds ratio (OR), risk difference (RD), hazard ratio (HR) for time-to-event. Pool on the log scale for RR, OR and HR because the log is approximately normal; back-transform for presentation.
• RR and OR are more consistent across baseline risks; RD is clinically direct but varies with baseline risk. OR exaggerates relative to RR when events are common, so never read a pooled OR as an RR.
• Continuous outcomes: mean difference (MD) when all studies use the same scale and unit; standardised mean difference (SMD — Cohen's d, bias-corrected Hedges' g, Glass's Δ) when scales differ; ratio of means for skewed data. Rough SMD benchmarks: 0.2 small, 0.5 moderate, 0.8 large.
• Proportions/prevalence: pool after Freeman–Tukey double-arcsine or logit transformation to stabilise variance near 0 and 1.
• Rates: log incidence rate ratio with variance from events in each arm.
• Sparse data / zero cells: add 0.5 continuity correction, or better use Peto OR (fine for balanced arms, small effects) or Mantel–Haenszel with no correction; exact/beta-binomial methods for very rare events.

Fixed-effect versus random-effects pooling

• Common (fixed) effect model: assumes one true effect; study estimates differ only by sampling error; weight = 1/variance (inverse-variance), so large studies dominate.
• Random-effects model: assumes true effects vary across studies around a mean, with between-study variance τ²; weight = 1/(variance + τ²), which flattens the weights, widens the CI and gives small studies more influence — so it is more vulnerable to small-study bias.
• τ² estimators: DerSimonian–Laird (closed form, classical, examinable, biased downwards with few studies), REML, Paule–Mandel, Empirical Bayes, Sidik–Jonkman. With fewer than about 5–10 studies the CI from DL is too narrow; the Hartung–Knapp–Sidik–Jonkman adjustment (t distribution) is now recommended.
• The random-effects CI is the precision of the mean effect; a prediction interval shows the range in which the effect in a new setting is expected to lie and is the honest summary when heterogeneity is real.
• Mantel–Haenszel is the default in RevMan for binary outcomes with few events; inverse-variance is used for continuous and for pre-computed estimates.

Heterogeneity, and how to investigate it

• Clinical (participants, dose, setting), methodological (design, risk of bias, outcome definition) and statistical heterogeneity.
• Cochran's Q (chi-square, k−1 df) tests the null of homogeneity; it is underpowered with few studies and over-sensitive with many, so never conclude "no heterogeneity" from p > 0.05 alone.
• I² = percentage of total variability due to between-study differences, not chance; Cochrane's rough bands 0–40% may be unimportant, 30–60% moderate, 50–90% substantial, 75–100% considerable. I² is not an absolute measure of the amount of heterogeneity — τ² and the prediction interval are.
• H² = Q/df and H = √(Q/df); τ is on the outcome scale and is the most interpretable.
• Investigate with pre-specified subgroup analysis (test for subgroup differences, not separate p-values), meta-regression (needs roughly ≥10 studies per covariate; study-level covariates only), sensitivity analysis (fixed vs random, high vs low risk of bias, different τ² estimators, including/excluding imputed SDs), leave-one-out and influence/baujat plots, and cumulative meta-analysis by year.
• Ecological (aggregation) bias: a study-level covariate in meta-regression does not estimate the individual-level relation — the key reason subgroup findings from aggregate data are hypothesis-generating only.

Publication bias and small-study effects

• Sources: non-publication of null results, time-lag bias, language bias, duplicate (salami) publication, citation bias, selective outcome reporting, funding-source bias.
• Funnel plot: effect (x) against precision (y, usually SE inverted); symmetry expected, a missing bottom-left/right corner suggests small-study effects. Needs ≥10 studies; asymmetry can also arise from true heterogeneity, poor methods in small trials or chance — asymmetry ≠ proof of publication bias.
• Tests: Egger's linear regression of the standard normal deviate on precision (intercept ≠ 0 = asymmetry), Begg–Mazumdar rank correlation, Peters' and Harbord's tests for binary outcomes (Egger is inflated for ORs with unbalanced data).
• Adjustments: trim-and-fill (imputes missing studies — a sensitivity tool, not a correction), selection models, PET-PEESE, limit meta-analysis; prospective trial registration (CTRI in India) is the real remedy. Failsafe N (Rosenthal) is obsolete.

Advanced designs and limitations

• Network meta-analysis: combines direct and indirect comparisons across a connected network; assumes transitivity and consistency (checked by node-splitting, design-by-treatment interaction, side-splitting); outputs league tables, relative rankings and SUCRA. Ranking statistics are often over-interpreted.
• Diagnostic test accuracy: pooled sensitivity and specificity must be modelled jointly (bivariate random-effects or HSROC) because they are negatively correlated through the threshold; present an SROC curve, not a single pooled ROC area from independent pooling.
• Trial Sequential Analysis / GRADE imprecision: repeated updating of a meta-analysis inflates type I error; TSA applies monitoring boundaries and a required information size.
• Limitations: garbage-in-garbage-out (pooling biased trials gives a precise biased answer), Simpson's paradox risk if arms are pooled across studies instead of comparing within study, unit-of-analysis errors (cluster, cross-over, multi-arm, multiple time points), double-counting overlapping cohorts, and reliance on reported rather than actual outcomes.
• Indian practice: Cochrane South Asia at CMC Vellore trains and supports review teams; HTAIn (Department of Health Research) commissions systematic reviews and cost-effectiveness analyses that feed PMJAY/AB-HWC decisions; ICMR's National Ethical Guidelines (2017) govern the underlying research, though a secondary review of published data generally needs only institutional/exempt review — confirm with the local IEC. Reviews drive NLEM and MoHFW/NCD guideline revisions, and WHO-SEARO guidance.

Key points

• Systematic review – pre-registered, reproducible search and appraisal; meta-analysis – the optional statistical pooling inside it.
• PICO – structures the question and directly generates the search strings and eligibility criteria.
• PROSPERO – prospective registration before screening; the main guard against outcome switching.
• PRISMA 2020 – 27-item reporting checklist plus the flow diagram, whose three stages are identification, screening and included (the 2009 "eligibility" stage was dropped).
• Duplicate screening and extraction – two independent reviewers, kappa reported, third reviewer for disagreements.
• RoB 2 / ROBINS-I / NOS / QUADAS-2 – study-level bias tools; AMSTAR-2 / ROBIS appraise the review itself.
• GRADE – certainty of the body of evidence: High/Moderate/Low/Very low, 5 downgrade and 3 upgrade reasons.
• Inverse-variance weighting – weight = 1/variance; precise studies count more.
• Common-effect vs random-effects – one true effect vs a distribution of true effects (τ²); random-effects CIs are wider.
• DerSimonian–Laird – method-of-moments τ²; use REML/Paule–Mandel and HKSJ CIs when k is small.
• Cochran's Q, I², H², τ² – heterogeneity test and measures; I² is a proportion, τ² is the magnitude.
• Prediction interval – expected effect in a future setting; the honest summary under heterogeneity.
• Forest plot – per-study effect, CI, weight, pooled diamond, heterogeneity statistics; funnel plot – small-study effects.
• Egger's test – regression test for funnel asymmetry; needs ≥10 studies.
• SMD (Hedges' g) – pooling across different scales; MD only when units match.
• Log scale – pool RR, OR and HR as logs, then exponentiate.
• Network meta-analysis – indirect comparisons under transitivity and consistency; IPD-MA – the gold standard for subgroups.
• Meta-regression – explains heterogeneity with study-level covariates; ecological bias means findings are hypothesis-generating.

Formulas and examples

Symbols: yᵢ = study effect on the analysis scale (log OR/RR/HR or MD/SMD), vᵢ = its variance, k = number of studies, CER/EER = control/experimental event risk. Effect measures are dimensionless; MD carries the outcome's unit (mmHg, kg, mg/dL).

Example 1 — Fixed-effect pooling of odds ratios (three trials of an ORS-plus-zinc protocol).

• Study A: OR = (20×70)/(80×30) = 0.5833; Var = 1/20 + 1/80 + 1/30 + 1/70 = 0.11012.
• Σw = 33.415; Σw·lnOR = −4.895 − 4.259 − 6.107 = −15.262.
• Pooled lnOR = −15.262/33.415 = −0.4567 → OR = e^−0.4567 = 0.63.
• SE = 1/√33.415 = 0.1730 → 95% CI: −0.4567 ± 1.96(0.1730) = (−0.7958, −0.1176) → OR 0.45 to 0.89; z = −2.64, p ≈ 0.008.
• Q = 9.081(0.0823)² + 7.209(0.1341)² + 17.125(0.1000)² = 0.062 + 0.130 + 0.171 = 0.36, df = 2, p ≈ 0.84 → Q < df so I² = 0%, τ² = 0: the random-effects result is identical here.

Example 2 — Heterogeneity and the random-effects model (four cohort studies, log RR supplied).

• Fixed: Σw = 195, Σwy = −58.268 → θ̂ = −0.2988 → RR 0.74 (95% CI 0.64–0.85; SE = 1/√195 = 0.0716).
• Q = 0.335 + 1.870 + 3.887 + 1.786 = 7.878, df = 3, p ≈ 0.049.
• I² = (7.878 − 3)/7.878 = 61.9% (substantial); H² = 7.878/3 = 2.63.
• C = 195 − 13,525/195 = 195 − 69.36 = 125.64 → τ² = (7.878 − 3)/125.64 = 0.0388 (τ = 0.197 on the log scale).
• Random: Σw* = 61.43, Σw*y = −17.890 → μ̂ = −0.2913 → RR 0.75, SE = 1/√61.43 = 0.1276, 95% CI (−0.5413, −0.0412) → RR 0.58 to 0.96 — same point estimate, much wider interval.
• 95% prediction interval: −0.2913 ± 4.303 × √(0.0388 + 0.01628) = −0.2913 ± 1.0101 → RR 0.27 to 2.05: in a new population the effect could be harmful, which the pooled CI alone hides.

Example 3 — Standardised mean difference (different depression scales). Intervention n₁ = 30, mean 24.5, SD 5.2; control n₂ = 28, mean 21.0, SD 4.8.

• s_p = √{[29(27.04) + 27(23.04)]/56} = √(1,406.24/56) = √25.111 = 5.011.
• d = 3.5/5.011 = 0.698; J = 1 − 3/(4×56 − 1) = 1 − 3/223 = 0.9865 → g = 0.689 (moderate).
• SE(g) = √{58/840 + 0.4747/116} = √(0.06905 + 0.00409) = 0.2704 → 95% CI 0.16 to 1.22 (excludes 0).

Example 4 — From pooled RR to NNT. Pooled RR = 0.747 for diarrhoeal hospitalisation; control (CER) risk in Indian district data = 30%.

• EER = 0.30 × 0.747 = 0.2241 → ARR = 0.30 − 0.2241 = 0.0759 (7.6 percentage points).
• NNT = 1/0.0759 = 13.2 → treat 14 children to prevent one hospitalisation. Note NNT depends on baseline risk: at CER = 5%, ARR = 0.0127 and NNT = 79.

Example 5 — PRISMA flow arithmetic. Records identified 1,842 → duplicates removed 412 → 1,430 screened → 1,297 excluded on title/abstract → 133 full texts retrieved → 108 excluded (62 wrong population, 29 wrong comparator, 17 no usable outcome data) → 25 studies included in the review, 19 in the meta-analysis (6 reported medians/IQRs without extractable SDs and went into the narrative synthesis).

Where it is used

• Guideline and policy development: ICMR and MoHFW guidelines, NLEM revisions, NTEP/NCD programme guidance, PMJAY Standard Treatment Guidelines, WHO-SEARO technical documents.
• Health technology assessment: HTAIn (Department of Health Research) and its regional resource hubs pair systematic reviews of effectiveness with cost-effectiveness models to inform package rates and procurement.
• Academic and dissertation work: MPH/MD theses, Cochrane reviews led from Cochrane South Asia (CMC Vellore), PGIMER, AIIMS, ICMR-NIE Chennai and the IIPHs; prevalence meta-analyses of anaemia, hypertension, TB treatment outcomes and mental-health burden are a large Indian literature.
• Pharma, CRO and medical-communications roles: indirect-treatment-comparison and NMA dossiers, evidence synthesis for regulatory submissions and value dossiers, systematic literature reviews for safety signals.
• Job roles: research associate/scientist-B at ICMR institutes, HTA analyst, evidence synthesis specialist, epidemiologist in NGOs (PATH, Jhpiego, Clinton Health Access Initiative), medical writer, biostatistician; library/information-specialist roles for search strategy design.
• Programme evaluation: state health departments and NHSRC use rapid reviews to decide on interventions (e.g. community health worker incentives, screening intervals) before scaling.

Interview questions

• Q: How is a systematic review different from a narrative review? A: A systematic review has a registered protocol, pre-specified eligibility criteria, an exhaustive and documented search, duplicate screening and formal bias assessment, so it is reproducible; a narrative review selects studies at the author's discretion.
• Q: Must every systematic review include a meta-analysis? A: No — if the studies differ too much in population, intervention, outcome or design, pooling creates a precise but meaningless number, and a structured narrative synthesis (SWiM) is correct.
• Q: What does I² = 80% mean? A: About 80% of the observed variability in effect estimates reflects real between-study differences rather than chance; it does not say how large those differences are, which is what τ² and the prediction interval tell you.
• Q: Why is Cochran's Q not enough to rule heterogeneity in or out? A: It is underpowered with few studies (a non-significant p does not mean homogeneity) and almost always significant with many studies, so it must be read alongside I², τ² and the forest plot.
• Q: When would you prefer a random-effects model? A: When the studies are clinically or methodologically diverse so a single true effect is implausible — the norm in public health; but with few studies use REML or Paule–Mandel for τ² and a Hartung–Knapp CI.
• Q: Why do we pool on the log scale for odds and risk ratios? A: Ratios are bounded below by zero and skewed, while their logs are approximately normal with a tractable variance, so the normal-based weighting and CI are valid; the result is exponentiated back.
• Q: Funnel plot asymmetry — does it prove publication bias? A: No. It signals small-study effects, which may come from publication bias, genuine heterogeneity, poorer methods in small trials or chance; it also needs at least about 10 studies to be interpretable.
• Q: What does GRADE add beyond risk of bias? A: It rates certainty for the whole body of evidence per outcome, downgrading for risk of bias, inconsistency, indirectness, imprecision and publication bias, and upgrading observational evidence for large effects or dose–response.
• Q: Why should PROSPERO registration happen before screening? A: It fixes the question, outcomes and analysis plan in advance, so selective outcome reporting and post-hoc subgroup hunting in the review itself can be detected.
• Q: Two trials report depression on HAM-D and PHQ-9 — how do you pool them? A: Convert each to a standardised mean difference (Hedges' g) using the within-study pooled SD, after checking both scales point in the same direction; a plain mean difference would be invalid.
• Q: A subgroup analysis shows benefit only in men. How much weight does it carry? A: Little unless it was pre-specified, the formal test for subgroup difference is significant, and there are enough studies; study-level subgroup results are open to ecological bias and are hypothesis-generating.
• Q: Why can't you pool sensitivity and specificity separately in a diagnostic review? A: They are negatively correlated through the diagnostic threshold, so independent pooling biases both; a bivariate random-effects or HSROC model is required.
• Q: What would make you run an IPD meta-analysis instead? A: When subgroup effects, time-to-event re-analysis, consistent outcome definitions or adjustment for the same covariates matter — it needs data-sharing agreements and far more time.

Common mistakes

• Treating a non-significant Cochran's Q, or an I² of 0% with four studies, as proof of homogeneity, and then reporting a fixed-effect CI that is far too narrow.
• Reporting a pooled random-effects CI with high I² and no prediction interval, so the summary looks far more transferable than the evidence allows.
• Pooling risk ratios or odds ratios directly instead of their logarithms, or interpreting a pooled OR as a risk ratio when the outcome is common.
• Unit-of-analysis errors: counting a cluster-randomised trial by individuals without an ICC-based design effect, double-counting the shared control arm of a multi-arm trial, or including two publications of the same cohort.
• Pooling mean differences across different instruments, or imputing SDs from medians and IQRs without a sensitivity analysis.
• Running Egger's test on 4–6 studies, or treating trim-and-fill output as the corrected effect.
• Single-reviewer screening and extraction with no kappa, and a PRISMA flow diagram whose numbers do not add up.
• Averaging quality scores and excluding studies on a cut-off instead of using domain-based RoB 2 judgements in a sensitivity analysis.

Related topics

Epidemiology: principles, measures and study designs · Advanced epidemiology: bias, confounding and effect modification · Biostatistics for health sciences · Causal inference basics · Multiple regression · Health technology assessment and economic evaluation