Dishonest Statistics#
You’ve probably heard the saying: “Never trust a statistic you didn’t fake yourself”. I don’t belief that’s particularly great advice. Instead of blindly doubting (or trusting) any statistic, the best approach is to make an effort to understand them.
There are far more people who misuse statistics out of ignorance than those who fake them on purpose. In other words, the problem isn’t just about the statistic being right or wrong — it’s about understanding what the results actually imply without misrepresenting them. A correct statistic with a misleading interpretation can be just as harmful as a fake one, if not worse.
The field of Statistics covers a huge range, from what data is measured to how results are interpreted. A statistic can be technically true while still being deeply misleading. Every analysis has a specific context, shaped by choices made along the way. Often, errors don’t come from bad intentions but from missing the fine print. Sometimes, an analysis provides the right answer — to the wrong question. So yeah, “Never trust a statistic you don’t know the context of”.
Flaws in Analysis#
These are the easiest errors to spot with a basic statistical knowledge. When reading a result is followed by a “wait, what?”, especially when the choice of statistical test or data transformation does not seem to make sense, you are encouraged to ask about more context. A statistical analysis shouldn’t be fancy, it should be the easiest and simplest way to answer a question correctly.
A Fisher’s exact test confirms that eating 3/4 of a pizza is not significantly different from eating 1/4, but cutting the pizza in 12 will make eating 9/12 slices significantly more than eating only 3 (p = 0.039).
Flaws in Measurement#
These happen very often, unfortunately, and point to a deeper problem in behavioral science. When the data collected doesn’t actually match the question being asked (see measurement error), any interpretation is problematic. Abstract concepts or latent variables such as intelligence are very difficult to measure, and we could fight about the measurement for hours before any statistics.
Our intervention significantly increased overall life satisfaction, with a 90% of participants reporting “seeing the value of a healthy lifestyle” on a single item scale after watching a 10 minute documentary on climate change.
Other flaws in measurement occur when the data collected is perfect, but the sample the data was collected from doesn’t represent the full population you are asking questions about (see selection bias). Mostly when measuring on a WEIRD sample and then trying to make it true for everyone.
Over 90% of people (who responded to our survey) report not to dislike being surveyed.
Flaws in Inference#
These are trickier because they require you to understand the context and assumptions made during analysis. Even with good data and a perfect analysis, results can still be misinterpreted. For example when failing to consider confounding variables, extrapolating averages to individuals, or assuming causation.
Men, on average, perform better on mental rotation tasks and navigation. So every man should outperform every woman, without even needing a map!
The Science behind this is also real, but whoever is in charge of interpreting the results often try to make the actual research a little more general, wouldn’t it be nice if it applied to everyone.
Chocolate consumption per capita is positively associated with the number of Nobel laureates in a country. Feed us more chocolate!
Or in this case just humorously constructed to make a point.
A staggering 100% of people who confused correlation with causation died. Be careful with statistics!
To keep you on your toes and practice critical thinking go check the research featured in the Ig Nobel Prize winners’ list. They are a great reminder that even the most serious research can be flawed, or that the most meticulous research can be blown out of proportion.