There aren’t many statistician jokes, I imagine, but here’s the only one I know.
Three statisticians go out on a hunting trip. After a while they see a deer. The first shoots and misses to the left. The second misses to the right. The third shouts “we got him!”
The point? Averages, even when true, are a mathematical fiction rather than a concrete reality—the notion of 2.4 children as the average nuclear family illustrates this well. We should bear this in mind whenever we use numbers to summarise people or experiences. Let me give you two examples.
Customers never have an average experience
Twice recently, when talking about excellent companies with a strong (and deserved) reputation for customer satisfaction, I’ve had the wind taken out of my sails by people objecting “well I think they’re rubbish!”, and they go on to explain exactly why. The thing is we’re both right, but I’m talking about an excellent average, and perhaps one or two even better highlights from conferences and case studies, while the other person is talking about their own unique experience. Even the very best companies get it wrong sometimes.
The lesson here is that, even if you are very good, it’s worth addressing the occasions when it goes wrong. In particular it’s a good idea to keep working hard on making sure you are as close to flawless as possible on the “givens”. Tracking percentages of low scores, as well as averages, is a good way to do this.
I am not a number
Where averages are really dangerous is when they are used to draw conclusions about groups of people (here we are a whisker away from statistically-backed bigotry if we’re not careful). The problem, really, is that statisticians know what they mean when they say “men are better than women at X” or “girls are better than boys at Y”. They mean, usually, that there is a small but statistically significant difference in the averages for the two groups. The problem is that’s not how normal people interpret the statement—which, at best, would be that almost all girls are better than almost all boys at Y.
Let’s look at an example I cooked up. Say we have administered a test to 1,000 girls and 1,000 boys, and they all have a score.
The analysis shows that girls have a higher average score than boys, and we can run a t-test that shows this is a highly statistically significant difference. Confidence intervals tell us the same thing. In other words girls are better than boys at this. But what if we get away from the averages and take a look at the distributions underlying them, i.e. get back to the raw scores?
The lesson is that comparing averages, though it can be useful, tells us very little about any particular individual. Next time you see a story of this pattern in the paper (or in books that make a fortune using tiny laboratory differences to back up cliches about empathy and reading maps), ask yourself what the underlying distribution looks like.