Imagine you asked ten friends living in different parts of the country each to toss a coin twenty times and tell you how many times it came up tails. Perhaps it is your friend in Scotland who gets the most tails. Would you therefore conclude there was something special and different about the results of coin tosses in Scotland from the rest of the UK? Of course not; the result was just the luck of the draw and if you did it again, it may very well be another part of the country that came top with the most tails. To be pretty sure that there really was a difference, you’d have to do something so many times that you could discount normal random variation as a cause.

That’s the key difference between random variation and statistical significance.  When researches say something is “statistically significant” they mean that the variations in results are beyond those you could reasonably expect to be caused by pure chance and so you can begin to draw some conclusions about cause and effect. It’s a common issue with opinion polls and similar surveys: ask lots of people around the country the same question and you’ll get some random variation in the results across the country even if there isn’t any actual underlying geographic pattern. It’s only if you get enough variation that you can begin to say, “People in Scotland are more likely…” or “People in Devon are less likely…”.

And all this brings us to the BBC’s coverage earlier this week:

The most sparsely populated county in Wales has been revealed as the happiest place in the UK.

One problem with that. As the BBC reported elsewhere, the researchers behind these figures had themselves said:

The researchers stress that the variations between different places in Britain are not statistically significant.

Even that comment though was half-way through a report which started:

The most sparsely populated county in Wales is where you will find Britain’s happiest place, say researchers.

Powys tops the list of 273 districts, with Edinburgh apparently the most miserable place in Britain.

Eight of the top 10 districts with the highest levels of wellbeing are in Scotland or the north of England.

All of which again is not what the researchers had actually said and leaves for a rather bizarre report. I mean, how often do you read a report which half-way through says, hey all of what we’ve just said is nonesense?

In other words, researchers find some random variation, say you can’t draw any conclusions from it about which are the happiest or saddest parts of the country and the BBC then report, err, which are the happiest or saddest parts of the country.

But it gets worse. Because not surprisingly various people have complained to the BBC about this misreporting. And the BBC’s response? Not to apologise for running the story. Not to try to defend the conclusions as being true after all. But instead to dismiss the complaint because:

We felt that as the story was of a light-hearted nature, and that as the conclusions were not of great importance, or significance, it made for a light and entertaining read.

There you have it: if a story is amusing and trivial, it doesn’t matter if it isn’t true.

Hat-tips: Bad Science and Gimpy’s Blog.