Randomised Controlled Trials in Public Policy: A Primer

#The problem randomisation solves

Comparing people who joined a programme with people who did not almost never isolates the programme’s effect, because joiners differ from non-joiners in motivation, health, income and countless unmeasured ways. This is selection bias, and it contaminates naive before–after and participant–non-participant comparisons alike.

Randomisation makes the control group a credible counterfactual: an estimate of what would have happened to the treated group without the programme. No amount of statistical adjustment of observational data can guarantee this, because adjustment only handles the differences that were measured.

#Landmark trials in public policy

• RAND Health Insurance Experiment (United States, 1971–1986). Families were randomly assigned to health-insurance plans with different levels of cost-sharing. Higher cost-sharing reduced use of care substantially, with little measured effect on average health for most participants — though with adverse effects among poorer participants with existing conditions. It remains the reference point for insurance-design debates.
• PROGRESA / Oportunidades (Mexico, 1997– ). A conditional cash-transfer programme rolled out with randomised phase-in across villages. Evaluations found increased school enrolment and improved child health, and the programme’s published, independent evaluation became a template copied across dozens of countries.
• Oregon Health Insurance Experiment (United States, 2008). Oregon allocated scarce Medicaid places by lottery, creating a natural RCT. Coverage increased health-care use, reduced depression and virtually eliminated catastrophic medical expenses; over the first two years it produced no statistically significant improvement in several measured physical outcomes — a nuanced result that both sides of the US coverage debate learned from.
• Moving to Opportunity (United States, 1994–1998). Housing vouchers enabling moves to lower-poverty neighbourhoods were randomly assigned. Long-run follow-ups found substantial adult earnings gains for those who moved as young children, reshaping research on neighbourhood effects.

The method’s spread beyond rich countries was recognised when the 2019 Nobel Memorial Prize in Economic Sciences went to Abhijit Banerjee, Esther Duflo and Michael Kremer “for their experimental approach to alleviating global poverty”. Networks such as J-PAL (founded 2003) have since run hundreds of policy trials worldwide.

#Reading an RCT result

1. Intention-to-treat. Good trials compare groups as randomised, whether or not everyone complied — preserving the benefit of randomisation and estimating the effect of offering the programme.
2. Uncertainty. Effects come with confidence intervals; “no statistically significant effect” in a small trial is not evidence of no effect.
3. Pre-registration. Trials that state outcomes and analyses in advance are far less vulnerable to selective reporting.
4. Outcome choice. Check what was actually measured, over what horizon; many programmes plausibly affect outcomes trials were too short or too small to detect.

#What RCTs cannot do

• Generalise automatically. An RCT estimates the effect in its setting, population and era. Transporting results requires argument or replication — the external-validity problem.
• Capture system-wide effects. Small trials miss general-equilibrium consequences: a job-search programme that helps participants may partly displace non-participants.
• Answer value questions. Trials estimate effects; they do not decide whether those effects justify costs or override competing aims.
• Always be feasible or ethical. One cannot randomise constitutions, pandemics or central-bank policy; for many questions, quasi-experimental methods are the realistic ceiling.

#When randomisation is impossible

Where trials are infeasible, researchers exploit natural experiments — policy discontinuities, lotteries, staggered roll-outs — using designs such as difference-in-differences, regression discontinuity and instrumental variables. The 2021 Nobel Memorial Prize (David Card, Joshua Angrist, Guido Imbens) recognised precisely this toolkit. The logic is the same as an RCT’s: find variation in exposure that is plausibly unrelated to the outcome, and defend that assumption openly.

#Sources and further reading

• J-PAL (povertyactionlab.org) — trial registries, evaluations and teaching resources on randomised evaluation.
• RAND Corporation — the Health Insurance Experiment archive and summaries.
• Finkelstein et al., the Oregon Health Insurance Experiment papers (Quarterly Journal of Economics, 2012; New England Journal of Medicine, 2013).
• Cochrane (cochrane.org) — the systematic-review organisation whose evidence standards grew from clinical trials.