Skip to content

The Chance Result the Whole World Yearned To Believe

Last updated on May 28, 2015

My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded.

From a fascinating article with the click-baity title “I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How,” by John Bohannon on io9.

“But wait,” you say, “if this was all based on actual data and the finding wasn’t false, why are the findings meaningless?” Because of this:

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.

If you read this blog regularly, you already know this, but for those of you who might be new around here, the bottom line is this: Just because a claim is scientific doesn’t mean it’s true. I would add: This is doubly so for nutritional claims, and triply so for nutritional claims that actually sound fun.

ht: Jason Kerwin and Maeve Gearing, via Facebook.