Five Questions to Ask Before Trusting a Statistic

Few people have time to audit every number they encounter; fortunately, most bad statistics fail quickly under a handful of standard questions. The five below cover the failure modes we encounter most often in public debate.

#1. Who produced it, and can I find it?

A statistic without a findable source is a rumour with digits. Trace the number to its producer — a statistical office, a named survey, a published study — and check that the producer actually says what is claimed. Numbers frequently mutate as they pass through headlines and social media.

#2. How is the thing defined?

“Unemployment”, “poverty”, “migrant”, “excess death” — every measured concept has a technical definition that may differ sharply from the everyday word. Definitional choices routinely move headline numbers more than real-world change does; we examine this at length in Why Definitions Decide Debates.

#3. What is the denominator?

Raw counts grow with population; “a record number of X” may mean nothing but a record number of people. Ask: per how many? Compared with the relevant base? A risk that doubles from one in a million to two in a million has doubled — and is still tiny. Percentages of small or shifting bases mislead in both directions.

#4. Where did the uncertainty go?

Estimates come with margins of error and confidence intervals; headlines strip them. Small movements inside the margin of error — common in polls — are reported as swings. If a source publishes no uncertainty at all, treat precision as false.

#5. Compared with what?

Every “more”, “worse” or “record” embeds a comparison. Check the baseline: is it the relevant period (or a cherry-picked trough)? Are the units and definitions the same on both sides? Cross-country comparisons in particular routinely compare numbers produced by incompatible methods — a pitfall our review of trust surveys documents in detail.

A number that passes all five checks may still be wrong — but it has earned provisional trust. A number that fails any of them has not earned repetition.