Customer Satisfaction Measurement - The Leadership Factor

Research...it's not brain science

The New York Times has received a bit of a roasting over this piece (published a couple of weeks ago) using brain imaging (fMRI) to draw conclusions such as:

When we showed subjects the words "Democrat," "Republican" and "independent," they exhibited high levels of activity in the part of the brain called the amygdala, indicating anxiety.

This is nonsense. More to the point, it's such obvious nonsense that the piece should never have made it to print, which would have saved the NYT the embarrassment of this scathing response and an enthusiastic pile-on from the blogosphere—Ben Goldacre's piece was where I picked up the story, and Language Log has a guest post from the astonishingly distinguished Martha Farah thoroughly eviscerating the original.

Neuroscience is fascinating, but it's also particularly prone to abuse and pseudoscience. Merely showing pictures of the brain to people has been shown to make them more prone to accept flawed explanations...in other words "brain scans indicate" is much more persuading than "researchers think". Even though that's basically the same thing. The research behind this is summarised in a Language Log post, or you can read the journal article[PDF] in press.

All of which should make us very sceptical whenever someone claims to have done anything useful with "neuromarketing". Interestingly, the best blog I know of that claims to deal with neuromarketing has precious little brain imaging, but rather a lot of well-designed traditional experiments. We're not going to be stopping people in the street and asking them to stick their heads in a fMRI scanner any time soon.

Which, on balance, is probably a good thing.

Another paradox?

Sean posted an interesting comment in response to my post about Simpson's paradox. His question was

How about another research paradox.You improve in all requirements that are important to customers, significantly in the priorities for improvement, and as expected the satisfaction index increases.However when we look at how the customer rates their overall experience this has not changed.

So the conundrum is: how can satisfaction scores go up for a list of factors that customers say are important to them, but stay the same for an overall satisfaction question?

The first thing to say is that this is very, very, unusual. If you look at the relationship between overall satisfaction and Satisfaction Index there is typically a very high correlation between the two (in the region of .8-.9).

Let'€™s put together a list of potential explanations, and see which ones look most likely.

The way overall perceptions are reported

Overall questions are often reported differently to individual satisfaction scores and the Satisfaction Index. If this question is reported as a percentage "top box" score (and/or bottom box) then it will behave differently to an average, making it more volatile but also less sensitive to some types of change (a top box score would be blind to the elimination of poor performance, for example). It's measuring different things.

As an experiment I looked at the average overall satisfaction score, average Satisfaction Index and % top box on overall satisfaction by month for one client with a very large set of tracking data (42 months). The correlations were:

• Average overall satisfaction with average Satisfaction Index: .86
• Top Box overall satisfaction with average Satisfaction Index: .07

In other words average overall satisfaction is very strongly correlated with Satisfaction Index, but % Top Box is very poorly correlated.

Try looking at the average overall satisfaction score as well as a top box figure to make sure you're getting a fully rounded view.

Sampling error

If you're talking about the change between one survey and another, rather than a consistent trend over time, the issue may be sampling error. Whenever you conduct a survey there is a certain amount of sampling error reflecting the fact that you only talk to a subset of the people whose views you're interested in. This gives every piece of data a margin of error, which will be relatively larger for a single question like overall satisfaction than it is for a composite score like the Satisfaction Index.

Sampling error may be hiding real gains in overall satisfaction by making the score look higher than it should in the first survey and lower than it should in the second.The Satisfaction Index is more resistant to sampling error.

Asymmetric impact

Nigel talks about this quite a lot in his new book. If improvement has mainly been in areas that are "satisfaction maintainers" then you wouldn't expect to see much change in average overall satisfaction, since the shape of the relationship between a given and overall satisfaction is a bit like this:

What you would expect to see is a reduction in the number of customers who are very dissatisfied overall, as the key with givens is to keep performance above a threshold "tipping point".

So is this the answer? Perhaps partially, but it's unlikely that all the priorities for improvement are givens. (It'd be nice to know the impact correlations!)

Improvements to satisfaction maintainers may not result in big changes in average overall satisfaction.

The nature of overall satisfaction

Another possibility is that customers have noticed improvements in specific areas, but this has not yet had time to feed through to their overall feelings about the organisation. If so, we'd expect to see overall satisfaction start to trend up (along with loyalty) after enough time has elapsed. This could be confirmed with statistical models given enough data.

A related explanation is that overall satisfaction is a tricky beast. One of the reasons that the Satisfaction Index is often preferable to a single overall satisfaction question is that it is rooted in specific attributes. Overall satisfaction is a much more nebulous measure, and is bound to incorporate many other influences such as brand image and reputation.

This helps explain why there may be a time lag between improvement in individual attributes and changes to overall satisfaction. It takes much longer for customers to feel generally warmer about an organisation than it does for them to notice specific improvements.

Conclusions

So what is the most likely explanation for Sean's paradox? My guess is that it is probably down to looking at improvements in average scores against a relatively static % top box on overall satisfaction.

If I'm wrong, and Sean's talking about an average overall satisfaction score, then my guess would be a mixture of some of the other issues I've outlined in this post. With a bit more digging around in the data we might be able to eliminate some of them and narrow down the search!

ACSI - Consumer Satisfaction Takes A Dip

The ACSI (American Customer Satisfaction Index) has reported that Customer Satisfaction is down for the first time since early 2005. This is one of the findings in their third quarter report. The index has slipped 0.1% to 75.2 on its 100 point scale, but it is still higher than it was a year ago.

The report makes interesting reading and picks out a few winners and losers amongst the different brands and sectors. For those of you in the UK, it's an interesting exercise to compare these results to the findings of the UKCSI.

It gets measured therefore it's still getting done

Enterprise is widening the gap from its competitors in satisfaction
Nov 2007

Back in October 2004, Stakeholder Satisfaction featured an article on how customer satisfaction was treated at Enterprise Rent-a-Car. The article was entitled "What gets measured gets done" and it explained Enterprise's internal measure of satisfaction "ESQi" or "Enterprise Service Quality Index".

The Enterprise way is apparently still bringing in results, as a recent survey quoted in American Business Travel News shows. To quote:

For the fourth year in a row, Enterprise Rent-A-Car was the highest-rated car brand in the study.

A key point in Enterprise's success has been using the right measures to effectively manage employee's behaviour and organisational performance. Hill et al even cite Enterprise as a prime example of why you should measure satisfaction (see page 35 of Customer Satisfaction).

The elephant in the room

A mini rant this time, on a topic that's usually swept under the carpet.

Delegates on our training courses often get confused by the term "sample" because the same word is used for both the number of questionnaires you send and the number you get back. Why? Because in the examples given in most textbooks, the two numbers are the same.

If you turn to an introductory statistics or research text, the tacit assumption in almost every case is that the sample achieved is 100% of those who were invited. These books then skip merrily on to statistical estimation of confidence intervals, margins of error, significance tests and so on. Fun!

In practice much the same thing happens—analysts happily apply the formulas given in those textbooks to their data. Formulas that tell them about the accuracy of their data and their ability to measure things out there in the real world.

Great, except for one thing—response rates aren't really 100% are they? Which means you may have a problem with non-response bias. It also means that the standard statistical formulas cannot give a perfect measure of how precise your data are. In theory, if you don't get a near perfect response, you can't project the findings of your survey to the population you're interested in.

So what do we do? We get the response rate as high as we can and hope for the best. But getting a good response is really, really important. So much so that I would argue response rate is actually more important than sample size.

All of which boils down to a very simple suggestion—sometimes it would be worth sacrificing sample size for a better response rate. In other words, if you are suffering from a lack of response it would be better to use a more expensive, but more effective, technique like telephone or even face to face interviewing rather than self-completion methods.

Bizarrely this will give you less apparent reliability, since the formulas don't take response rate into account, but it will immensely improve the robustness of your survey. As a rule of thumb aim for at least a 40% response rate, preferably 50%, and don't pretend you can't see the elephant in the room.

76% of Surveys are 'Voodoo Polls' (or are they?)

Has the proliferation of of DIY survey tools improved the standard of surveys that we see in the media? Probably not seems to be the answer. Seth Godin's recent post on surveys raises some interesting points. He identifies four kinds of survey:

- Census Surveys
- Public Surveys
- Professional Surveys
- Census-based Analytics

I think the type we see most in the media is the public survey which typically quotes a "58% of listeners think..." type of statement. Certainly, surveys are more and more popular in the media and as the basis of press releases. In itself, that isn't a bad thing. What has to be avoided though is confusing these voodoo polls* with a carefully structured, scientifically designed survey sent to a representative sample of your audience.

It's worth considering that although the necessity of basing the results of a CSM survey on a representative sample of customers is widely acknowledged, the technical aspects of doing so are little understood and often neglected, making the survey unreliable and not much better than a 'voodoo poll'.

*Voodoo polls is the term commonly used to refer to the voluntary sort of surveys used on the TV and radio - 'text in and let us know what you think'. This type of survey notoriously suffers from unrepresentative samples and is generally unreliable.