According to a survey, 95% of journalists don't care about truth.
Bold claim? Not really, because "according to a survey" is a get out of jail free card that (I'm fairly sure) prevents anyone from being able to sue me.
The sad truth is that no-one really believes what surveys reported in the press say, unless they happen to reinforce their own pre-existing beliefs. Why? Because the vast majority of surveys enthusiastically reported in the press are horribly misused.
Ben Goldacre of Bad Science is one of many bloggers to pick up on the BBC's monumentally stupid reporting of a survey about the happiest places in the country. The researchers made it perfectly clear that none of the differences are statistically significant. In simple terms that means that the differences they found may reflect random survey error rather than real differences. In other words, there is no story. The message here is "some places may or may not be happier than others. We don't know." Understandably, the BBC didn't think that was especially newsworthy.
Instead of reporting the real message, they decided to ignore the stuffy old statistics and just present the apparent differences as fact. This is, frankly, lying. When called on the issue, their response was to say that it's okay, because the piece is of a "light-hearted nature".
The problem is that every time this happens it erodes the basis on which survey research, and statistics more generally, rest. Statistical significance is not something we can ignore when it suits us, it is fundamental to knowing how much faith we can put in a statistic.
Events such as this are a powerful argument for replacing statistical significance, which can look stuffy and irrelevant, with confidence intervals. Going back to my survey at the beginning of this post, it won't surprise you to learn that this isn't a statistically significant finding. With a confidence interval of +/- 30% the real answer could be as low as 65%. But I doubt it.
We blogged on the difference between significance testing and confidence intervals back in March.