Customer Satisfaction Measurement - The Leadership Factor

Never trust an average

There aren’t many statistician jokes, I imagine, but here’s the only one I know.

Three statisticians go out on a hunting trip. After a while they see a deer. The first shoots and misses to the left. The second misses to the right. The third shouts “we got him!”

The point? Averages, even when true, are a mathematical fiction rather than a concrete reality—the notion of 2.4 children as the average nuclear family illustrates this well. We should bear this in mind whenever we use numbers to summarise people or experiences. Let me give you two examples.

Customers never have an average experience

Twice recently, when talking about excellent companies with a strong (and deserved) reputation for customer satisfaction, I’ve had the wind taken out of my sails by people objecting “well I think they’re rubbish!”, and they go on to explain exactly why. The thing is we’re both right, but I’m talking about an excellent average, and perhaps one or two even better highlights from conferences and case studies, while the other person is talking about their own unique experience. Even the very best companies get it wrong sometimes.

The lesson here is that, even if you are very good, it’s worth addressing the occasions when it goes wrong. In particular it’s a good idea to keep working hard on making sure you are as close to flawless as possible on the “givens”. Tracking percentages of low scores, as well as averages, is a good way to do this.

I am not a number

Where averages are really dangerous is when they are used to draw conclusions about groups of people (here we are a whisker away from statistically-backed bigotry if we’re not careful). The problem, really, is that statisticians know what they mean when they say “men are better than women at X” or “girls are better than boys at Y”. They mean, usually, that there is a small but statistically significant difference in the averages for the two groups. The problem is that’s not how normal people interpret the statement—which, at best, would be that almost all girls are better than almost all boys at Y.

Let’s look at an example I cooked up. Say we have administered a test to 1,000 girls and 1,000 boys, and they all have a score.

Gender	Mean	N	SD	Confidence Interval
Male	7.0	1000	1.4	+/- 0.1
Female	7.6	1000	1.5	+/- 0.1
Total	7.3	2000	1.5	+/- 0.1

The analysis shows that girls have a higher average score than boys, and we can run a t-test that shows this is a highly statistically significant difference. Confidence intervals tell us the same thing. In other words girls are better than boys at this. But what if we get away from the averages and take a look at the distributions underlying them, i.e. get back to the raw scores?

I bet you weren’t expecting that much overlap, were you? About 35% of females have a score lower than the average for males, and about the same number of males have a score higher than the average for females.

The lesson is that comparing averages, though it can be useful, tells us very little about any particular individual. Next time you see a story of this pattern in the paper (or in books that make a fortune using tiny laboratory differences to back up cliches about empathy and reading maps), ask yourself what the underlying distribution looks like.

Snake oil business

Image source: http://www.flickr.com/photos/niosh/2492023651/

Everyone loves science. Researchers, businessmen, journalists, PR and marketing people, everyone. Why? Because science is all about clever people in white coats “proving” your pet theory. Nothing sells the latest consultancy fad better than a tame PhD, and what kind of fool would pay £3 for an own-brand moisturiser when you can get one with Unobtanium QZ for only £30? Science sells.

Except it's not science, is it? Just like the snake oil salesman of old, con artists have always enjoyed dressing up in the clothes of science to rip off the unwary. Wearing a white coat and using complicated-sounding words does not make you a scientist any more than wearing a tutu makes me a ballerina.

So what does? There are hundreds of possible answers, but I think a defining feature is the willingness to be proved wrong. The ladybird version of science is that someone has a clever idea about how the world works, and then thinks of a way to test if the idea is true or not. The second part is the important bit—good ideas are ten a penny, but most of them are wrong.

Human beings are not very good at thinking like this. We are susceptible to something called confirmation bias, which means that we tend to notice information that confirms our preconceptions and discount the things that don't. So scientists have to be very careful to specify up front what their test is, and what different outcomes will mean for their idea.

Unfortunately, businesses are very reluctant to embrace the philosophy of science. Getting the authority of science by hiring PHDs and quoting figures to 5 decimal places is great, but the scientific method is hard. Worse still, facts have a distressing tendency not to tie in with the way we want the world to work.

So here's a challenge for you. Take your most cherished beliefs about your business and state them as hypotheses:

• Satisfied customers are more profitable
• Engaged employees have lower rates of absenteeism
• etc.

and devise a test for them. Now the hard bit, if the test doesn't support your hypothesis, throw it out and start again. If it does (actually even if it doesn't), tell the world. Congratulations, you are now a scientist—white coats are available here. 

The rebirth of induction

Interesting post (content warning if pictures of brains disturb you) by Jonah Lehrer, whose book—The Decisive Moment—is well worth reading after you've finished Predictably Irrational. He argues that the industrialisation of science may mean that scientists move from being associated mainly with conducting experiments back to a greater emphasis on finding theories to explain masses of data.

This echoes the idea that, increasingly, we will be relying on data mining and machine learning in place of traditional sources of data and statistical techniques. This Business Week article, for instance, has been incredibly popular (and is a good, if sightly dull, read).

I think these techniques show loads of promise, although it will be a long time before most businesses are able to get sufficient control over the masses of data they stack up to feed much of it to the data miners. There are also, probably more here than in the US, data protection and privacy concerns in terms of the way we use indivdiual's data.

Most importantly, we need to be able to distinguish between the ability to predict something and the ability to explain it. This is like the difference between understanding that the sunrise is an illusion caused by the Earth's rotation, and worshipping the cockerel as an avatar of the sun god.

10 tips for presenting data that will drive action

I take both a personal and a professional interest in presentation. Getting a presentation right means your audience will leave with the right message, understand what the data is telling them, understand what they should be doing, and be inspired to take action.

Presentation is key, and the ultimate test of success or failure is not a ripple of polite applause and a kind word from the Chief Exec, but whether your business changes as a result.

I believe that the main factor which determines the success of a presentation is the clarity of the message, both on individual slides and in the presentation as a whole. This is doubly true of numerical data, such as customer survey results, where it is easy to get bogged down in slide after slide of analysis.

When it comes to presenting numerical data, here are 10 key principles that I have stolen or evolved over the years. These form the basis for most of the recommendations I make on my course.

1. Show me the numbers
Simplicity comes from clarity of message, not dumbing down. If you hide the numbers then I have to trust you, and frankly I don't. By all means use a R2G colour scheme to make the message clearer, but numbers are not innately scary and it's important that they are there for those that want them.

2. Order by something
There is almost always an order more interesting than simply listing items alphabetically or in the way they were presented on the questionnaire. Think about the story the data is trying to tell and then sort it in a way that supports that (e.g. largest values first, largest changes first, largest differences from average first etc).

3. Let the data breathe
Get hold of the Visual Display of Quantitative Information, or something similar, and have your eyes opened to the amount of “chartjunk” and wasted ink in the average table or chart. Fade chart mechanics such as grid lines and axes into the background by changing them from black to grey. You will soon find that you're obsessing about allowing the data room to stand on its own, rather than decorating it. “If the numbers are boring then you've got the wrong numbers.”

4. Tell the truth
This might seem obvious, but it is easy to fool yourself as well as your audience, particularly when evidence can be made to conform to your preconceptions. Make sure you are asking yourself tough questions, and that you can back up your assertions with evidence. I recommend the Halo Effect as an effective antidote.

5. If it's boring, ditch it
When we do a piece of analysis, or ask a question in a survey, it is often because we're wondering if something interesting will emerge. Often it doesn't. If so, I recommend you leave that question or analysis on the cutting-room floor—padding out your presentation with analysis that has no implications for the future dilutes the message and makes you look dull. Have the courage to make decisions about which analysis is meaningful.

6. You are responsible for the actions that are taken
This is a scary one. Using the wrong piece of analysis, or using it in an inappropriate way, can lead to bad decisions. Quadrant charts are notorious in this regard. This can lead to real, damaging, effects in the way your organisation performs. Be careful, and make sure you are confident that you have got everything right. If you want people to take your analysis seriously, you have to take it seriously too.

7. Don't create work for your audience
There is no need to dumb down for your audience, but there is also no reason to be obstructive. Avoid abbreviations, jargon and other barriers. Subtler sins include using poor typography, such as words rotated by 90° in order to fit. This can almost always be avoided, and doing so is a sign of respect for your audience.

8. Analysis and reporting are separate functions
This links in with point 5. Not everything that happens at the analysis stage should end up in the reporting/presenting phase. This is upsetting, since you may want to demonstrate how much hard work you did, but crucial if you want to leave a clear message.

You may also choose to present messages in a different way, or using different charts, than the tools you used to discover them in the first place. A factor score coefficient matrix is unlikely to mean much to the board, but the message about the existence of common factors underlying the items on the questionnaire is perfectly comprehensible.

9. Delete first, then add
Perhaps the easiest way to achieve number 3 is to remove the unnecessary ink that Excel and other programmes tend to add to graphs. Clearing away the clutter, then fading the chart mechanics into the background, will free up lots of empty space. This is a good thing in itself, but it also allows you the room to add elements which help the message stand out. Don't be afraid to annotate directly onto your charts where it helps focus attention on the message you're trying to tell.

10. Find a message, then sell it
The single most important thing is that you know what the analysis is trying to say. A good discipline when preparing presentations is to write “headline style” titles. By using strong declarative statements (“Satisfaction is up in the North West”) rather than category headings (“Regional satisfaction vs. 2008”) you are forced to think about what the message is, as well as making a much more engaging presentation. If you can't find a headline, then see point 5.

Looking across these headlines should allow you to see a building story that will leave the audience in no doubt about what your main message(s) are. If not you need to decide what they are and restructure the presentation to deliver them clearly.

How scales work

Bear with me, this will get relevant!

The world of climbing has, somewhat oddly, been riven with controversy over scales. One of the UK's top climbers, James Pearson, made a first ascent of a very hard and dangerous new route which he named “Walk of Life”. So far, so good.

He decided to give the route a grade of E12 which, without getting into the technicalities of the British trad grading system, would make it the hardest climb in the world. Many armchair critics were sceptical about the grade, and the situation got even more heated when Dave MacLeod repeated the route and gave it a grade of E9 (still very hard and dangerous, mind you).

 

[Image source: http://davemacleod.blogspot.com/]

The poorly-informed and ill-tempered debate that followed in the climbing media (which both Dave and James rose above) focused on whether the grading scale is “broken”, or whether climbers give grades in order to attract publicity/make themselves look hard.

I think this is missing the point completely. The real lesson is that it is impossible for humans to give a completely objective measurement of anything. This is similar to the objection that “one person's 8 is another's 9” or, for that matter, “one person's satisfied is another's very satisfied”.

It's never a good idea to rely too much on a single score given by a single person, but the law of large numbers means that, over a sufficient number of people, average scores will stabilise and be reliable. This is the magic of quantitative surveying—averaging lots of unreliable scores gives us a reliable average.

Interestingly, the few sensible comments about the Wall of Life grading debate draw attention to the fact that climbing grades are always established by consensus. This is simply an informal version of the same phenomenon.

Analysing qualitative data

Delegates on my training course often ask me about analysing qualitative data, usually in the form of verbatim comments from customers. Compared to quantitative data, qualitative data often seem bulky and time-consuming—surely there must be an easier way?

Sadly there isn't, really. There is no software with a magic “code all this data” button, and even if there were it would be a drastically bad idea to use it. This is an area in which a human brain needs to be employed for any real insights to emerge. Mind you, so is quantitative analysis...but that's a post for another day!

There is software designed to make your life easier when it comes to analysing qualitative data. Collectively these programmes are known by the stunningly unwieldy acronym CAQDAS*. If you want a full list you can find one at the ASC website (caution—ugliest website ever). Prominent names include NVivo, Qualrus and The Ethnograph.

While they can't do the analysis for you, computer programmes do give you tools that make your life easier. You can get your computer to produce a word frequency count and concordance (key words in context), as well as to search for key phrases. But this kind of automated work is rarely very interesting.

CAQDAS software really comes into its own in easing the process of building up a coding frame and then using that to generate theory. Many CAQDAS programmes are built around an academic approach known as Grounded Theory, which is great for inductive reasoning in your PhD thesis but, frankly, probably overkill for your customer satisfaction survey.

Whatever software tools you use, effective analysis of qualitative data will always take time and care. Coding is a back-and-forth process that takes frustratingly longer than you expect. Above all you need to remember that coding can take you away from the depth of the underlying data—the codes are there to help you think, not to convert your data into arbitrary quantifiable categories!

* Computer Assisted Qualitative Data Analysis Software or sometimes QDAS or QDA for short!

Statistical significance

A small addition to yesterday's rant measured critique about the reporting of surveys.

Even when a result is statistically significant, it doesn't mean that we should draw lots of conclusions based on it. Why not?

• Some of the assumptions the statistical tests are based on may be violated (e.g. random sampling).
• Statistically significant results are not always meaningful - a difference may be real, but very small. Most stories along the lines of "men are better than women at..." fall into this category.
• 5% of differences which are statistically significant are not really there - we know that our tests have a 5% false positive error rate. If a result is particularly surprising then wait for replication before you leap to conclusions. "Extraordinary claims require extraordinary evidence" is not a metaphysical statement, it's all about the sums.

Anyone who has followed my posts about significance tests will have realised that I think there is one simple fix for most of these problems - use confidence intervals instead. You can still be wrong, of course, but confidence intervals make you think much harder about the realities of what you do and don't know.

"If you can keep your head when all about you are losing theirs..."

...then you must be a statistician.

We've blogged this topic before, but there's a lot more going on beneath the surface than we like to believe whenever we make a decision. It suits us to believe that we make sensible, rational, decisions. In practice our emotions often play a big role in decision making.

An interesting New Scientist article discusses the science behind this, and the troubling impact it can have on our evaluations of, for example, the comparative risk of terrorism and road accidents.

We discuss emotions in some detail in our Customer Emotions Briefing, a half day course looking at the science and practice of dealing with emotions and their role in consumer decisions. Our conclusions? Without giving too much away, some of the headlines are:

• Emotions are crucial in decision-making
• Both conscious and unconscious processes have a role to play
• Emotions are not necessarily irrational
• Cognition is not necessarily rational!

The New Scientist article, and a blog post based on it, also comment on the importance of understanding how people make decisions when it comes to presenting information to inform those decisions. They bring in research by Gerd Gigerenzer, a leading authority in this area, and an author well worth reading.

We tend to overestimate the chances of "dread risks", such as terrorism or cancer, while underestimating the chances of suffering from other accidents or diseases. Fear, and the graphic portrayals of such events in the media, outweighs the cold statistics on causes of death.

Is overreacting to scary, but rare, risks a problem? It is if we, for example, opt for potentially dangerous and invasive investigative surgery without good reason. Gigerenzer has a superb article on statistical illiteracy when it comes to medicine at the BMJ website. It's particularly good on the problem of sensitivity/specificity we blogged last week.

More importantly, it suggests that if we want people to draw the right messages, we need to think carefully about the way statistics are presented. We have to think hard to find ways to make the numbers clear to a non-expert audience. In addition, if we want them to drive decisions, they should be graphic and memorable. VoxPops, customer comments, and even cardboard cutouts are all effective ways to bring the numbers to life.

Peter Norvig on statistics and experiments

A site I keep coming back to is Peter Norvig's* excellent piece on warning signs in the design and interpretation of experiments and the statistics that back them up. This is well worth reading and absorbing.

Even if you're already familiar with all the traps he outlines, it's rare to see them explained so concisely and so clearly all in one place. Many of them are relevant to research in general, and customer satisfaction research in particular.

I especially like Norvig's very clear example of the care with which probabilities about tests have to be used to draw conclusions about the probability of an event ("Warning Sign 14"). Let's make it relevant to customer research. Imagine that we are a credit card issuer, and we know that 80% of customers who defect have not used their card for at least a month. By contrast only 10.5% of customers who are not intending to defect will go a whole month without using their card. All we have to do is set up a trigger so that we call any customer who has been inactive for a month and we should be able to significantly decrease defections.

So what are the chances that an inactive customer will defect? Assuming that the base defection rate is 5%, the chances are actually less than 30% that each customer we ring is about to defect. Why so low? because defection is a comparatively rare phenomenon—even with a very accurate test the true defectors are outweighed by false positives. Exactly the same is true, as Norvig shows, for medical screening tests for breast cancer etc. The logic is explained in the table below:

	Defector	Non defector	Probability of defecting if inactive
Inactive	4	10	4/(4+10)=28.6%
Active	1	85
Total	5	95

So the accuracy of the test in correctly identifying defectors and non-defectors (technically the sensitivity and specificity) is very good (80% and 89.5% respectively), but this does not mean that someone who is inactive has an 80% chance of defecting.

Now go and read the rest of the article!

* Peter Norvig is a computer scientist who works for Google. His articles on a range of subjects are worth reading if you're interested in clear thinking and lucid, witty, writing. His online resume gives you an idea of his background, and also of how good an employer Google is:

"Note to recruiters: Please don't offer me a job.  I already have the best job in the world at the best company in the world."

Customer emotions briefing

Last Wednesday Richard and I presented our Customer Emotions briefing in London. This half day session looked at the psychology of emotions and how they feature in decision making, the role of emotions in driving customer loyalty compared to rational evaluations and finally some tools to help you measure and manage all the drivers of loyalty.

The session managed to be both high brow:

and low brow:

with a whole spectrum in between.

We'll be running the briefing near Manchester in July, so if you're interested in customer emotions and you live up North (or couldn't make it to the London one) maybe we'll see you there?