Customer Satisfaction Measurement - The Leadership Factor

Customer emotions briefing

Last Wednesday Richard and I presented our Customer Emotions briefing in London. This half day session looked at the psychology of emotions and how they feature in decision making, the role of emotions in driving customer loyalty compared to rational evaluations and finally some tools to help you measure and manage all the drivers of loyalty.

The session managed to be both high brow:

and low brow:

with a whole spectrum in between.

We'll be running the briefing near Manchester in July, so if you're interested in customer emotions and you live up North (or couldn't make it to the London one) maybe we'll see you there?

Scatterplots—they're really not that hard!

The intermittently good blog Junk Charts has another post highlighting the danger of thinking that you can analyse the relationship between two variables simply by plotting them next to each other on a line chart.

The problem is that line charts really don't give you the ability to tell if there is a relationship or not, but your brain may well con you into believing that they do because of our tendency to see patterns (even if they're not there). Take this chart:

Is there a relationship between the two lines? When you've made up your mind click here to see a scatter plot of the same data. It's a lot easier to tell isn't it?

Why are people so afraid of scatterplots? They're one of the most useful basic tools in any analyst's toolkit, and they're really a very simple idea—essentially a scatterplot is just a map. If you don't feel comfortable with them perhaps it's time you learnt to be!

How do customer emotions impact your business?

Customer emotions are often overlooked when organisations think about the customer satisfaction levels they are getting, or how to improve customer loyalty. But, the evidence is that customer emotions play a key role in this area.

When the concept of customer emotion is addressed by a business, more often than not it seems that it is of the 'wow - delight the customer' variety, but is this really the emotional engagement that customers are looking for?

Well, you can find the answers to the above points and read an article that goes into much more depth in the most recent Stakeholder magazine (see my previous post for the link).

There's also a half-day briefing that's being led by two of my colleagues in London later this month. You can see full details of the customer emotions briefing and book your place by following this link.

Show me the data

Nicholas Bissantz has another great post over at his blog. He returns to a favourite theme of his (and mine)—simplifying data for greater clarity is all very well, but sometimes it's best to show the actual numbers.

Averages can hide a wide range of actual performance. With large data sets, like surveys, we address that with statistical tools such as variance, standard deviation and confidence intervals. But what do you do when dealing with data that isn't statistical?

Bissantz suggest graphing the data points as well as the average, a technique that I've found very useful with small samples:

His example makes the point nicely—three very different real-world situations that would produce the same average, and therefore the same conclusion for an unwary analyst.

Significance testing is soooo 20th century

I'm just in the middle of an excellent book called Beyond Significance Testing. It's one of an increasing number of works to point out the flaws in both the fundamental theory and conventional usage of the "classic" significance test.

Let's have a quick look at how significance testing works. Imagine we want to see if there is a significant difference between the Satisfaction Index for men and women in the situation below:

How do we know if it's significant? The first thing is to work out what question we're asking, which is "does this difference exist in the population as well as in the sample?".  Then we create a null hypothesis, which is that there is no difference between men and women. Finally we conduct a hypothesis test, a t-test in this case, which tells us the chances of finding a difference as big as this in the sample assuming the null hypothesis (that there's no difference in the population). This value, the p. value, conventionally needs to be less than 5% for us to reject the null hypothesis...telling us that there is very unlikely to be no difference between men and women in the population (note that it doesn't tell us how big the difference is!). Clear as mud? I thought so.

So if this approach has flaws, which it does, what should we do instead? The alternative is to look at effect size and margin of error. In other words, instead of asking our stats package for an obscure p. value, let's ask it how big the difference is likely to be in the population...which is what we're really interested in.

One quick way to do this is to plot the margin of error, or confidence interval, of the Satisfaction Indexes straight onto the chart. This has a number of advantages: it is much easier to explain and understand, it illustrates the uncertainty of the measurement, and it captures the size of the difference rather than a simple "significant" or "not significant".

A better test of a specific question like this is to work out the margin of error of the difference, which in this case is 3.1 ±1.8. In other words we can conclude with 95% confidence that the difference between men and women in the population is between 1.3 and 4.9.

Much more powerful, and much easier to understand, than null hypothesis significance testing isn't it?

Customer Retention at Netflix

There's a great article on the Wired website about the effort that Netflix (an online movie rental company in the US) has put into improving its Cinematch algorithm. This is the bit of their website that says "if you liked this movie we suggest these movies as well". A tiny percentage increase in this algorithm's accuracy means fewer angry customers (because they haven't had what they perceive as 'stupid' suggestion) and that in turns means improved customer retention and loyalty. Read the full article here.

A Happy New Year to all our Customers?

Economic gloom
The economy, at least the one portrayed by the media is apparently struggling. Runs on banks, house prices collapsing, doom and gloom on the high street, the pound falling, younger people worried about mortgages, older ones about pensions. I suppose that’s hardly surprising if the Prime Minister chooses a Work and Pensions Secretary who he believes to be “incompetent”. The UK Consumer Confidence Index1 continued to fall in December and is now down to 85 from a high of 110 in March 2005. But what does that mean? According to its sponsors, “The Nationwide Consumer Confidence Index measures the population's view of the current position and future prospects of the UK. The index takes into account the general economic situation, employment conditions and personal expectations of the months ahead.” In other words, Jo and Josephine Bloggs expect ‘things’ to get worse rather than better in the future. Well at least it proves that people really do watch television and occasionally still even read something in a newspaper. Of course, Jo and Josephine haven’t got a clue what’s going to happen to the economy, but we can hardly blame them for that since research has shown that on average professional economists, investors and bankers have no more success in predicting financial futures than a monkey making random selections.

You’ve never had it so good!
So where does this leave customers? In a great place actually. Objective analysis confirms that most Jo and Josephines have never had it so good. Employment levels are higher than ever. Unfilled vacancies abound. The December jobless figure was just over 800,000 – the lowest since 1975! Homeowners have seen the value of their chief asset treble in the last decade and, for most of them that value continues to rise. Yes, read my lips – in most areas house prices are still going up, by 16% in London in 2007, by 12% in Scotland and the UK average by 9.3%. Even the prediction for 2008 is for an average 3% increase2. True, a few thousand people in the City will have been devastated that their bonuses didn’t even reach 7 figures in December but the economic impact of that human tragedy won’t extend much further than the Ferrari, champagne and personal shopper sectors of industry. No, most customers have never had it so good.

It’s official - UK customer satisfaction up!
Further evidence comes in the form of the UK Customer Satisfaction Index3 conducted by The Leadership Factor on behalf of the Institute of Customer Service. Figures from the latest wave released this month show customer satisfaction in the UK overall moving up by 4% to 69.4%. In customer satisfaction terms, 4% is a big increase over 6 months. So how can that make sense amidst all this doom and gloom? Simple! If customers still have jobs and money they benefit from all the economic pessimism. Companies get worried about future profits, competition intensifies and giving customers a good customer experience to keep them becomes more important. The main factors driving the improvement in UK Customer Satisfaction Index are higher customer satisfaction with prices plus strong increases in core service areas such as keeping customers informed, delivering on time and treating people like valued customers.

Competition drives customer satisfaction
This shines through when we look at customer satisfaction within sectors. Customer satisfaction is highest where competition is strongest, in sectors such as retail, insurance and automotive. The easier it is for customers to switch, the harder suppliers have to work to give them a flawless customer experience, or they’ll simply go elsewhere. It’s also noticeable that some very competitive sectors have recorded the biggest gains in customer satisfaction this time around. As well as insurance, the sectors working hardest to improve their customer experience is utilities, albeit from a very low base.

Improving customer satisfaction
I’ve been immersed in customer satisfaction for over 20 years now and it never ceases to amaze me how long it’s taken for companies to take on board the evidence that customer satisfaction pays. After all, it was as long ago as 1986 that the American Consumer Association told us that it costs at least 5 times as much to win a new customer as to keep an existing one. What I’ve noticed over that time is that some organisations have been much more successful than others at improving satisfaction. Why is that? Fundamentally it’s entirely down to how much they want to do it. If it’s a really important corporate objective, driven hard from the top, and provided they act on the conclusions from accurate customer satisfaction surveys, companies always improve customer satisfaction and loyalty, sometimes by a big margin. By contrast, when organisations pay lip service to customer satisfaction, but really they have other priorities, they never deliver a consistently good customer experience and customer satisfaction and loyalty don’t improve.

Leaders in customer satisfaction
Look at the leaders in the UK Customer Satisfaction Index. John Lewis and First Direct, to name two sector leaders that have a fully deserved reputation for making the customer experience their top priority. In the UK Customer Satisfaction Index their customer satisfaction scores are 89% and 86% respectively. John Lewis is a massive 16 percentage points ahead of its sector average for customer satisfaction and First Direct 13 percentage points ahead of the average for other banks. Contrast this with the other end of the customer experience spectrum. Local Government has a sector average of 58% on the UK Customer Satisfaction Index, with no improvement this time around. Since some organisations like the local ambulance and fire services do very well on customer satisfaction, you can just imagine how poorly people rate their customer experience with the average local council.

Low customer satisfaction at Currys
Of course, apart from the fact that you’d think councils would have a duty to give their council tax payers a good customer experience, what penalty is there if they don’t? much harder to understand is how a company in a very competitive sector, like retailing for example, would allow itself to have really low customer satisfaction over a significant time. I’m thinking about DSG, formerly Dixons, and including Currys and PC World. Two years ago at the The Leadership Factor’s Customer Satisfaction and Loyalty Conference held a Stamford Bridge, I remember highlighting the massive growth in online shopping that Christmas and asking how bricks and mortar retailers like Dixons/Currys were going to compete in a market where one supplier’s Panasonic 42” plasma was identical to its competitor’s? Short of closing all the stores and becoming an e-tailer, the answer, of course, was to make the in-store customer experience so good that people would be happy to pay a little more in-store because the overall value was greater and they ended up more satisfied. In 2005 UK customers spent £19bn online, a 13-fold increase in 5 years. In 2007 it was up to £46bn and still rising. What have DSG done? Neither one thing nor the other it appears. They’ve invested heavily in on-line, without success. According to results released by Brand Republic this month, Play.com and Amazon have the highest customer satisfaction amongst UK online retailers at 76% and 75% respectively. Trailing well behind are PC World (59%) and Currys (60%), though neither is quite as bad as B&Q, which brings up the rear at 53%. Unfortunately for DSG, the customer experience in-store doesn’t seem to be any better. The 2007 Which? High Street Shops Survey placed Currys 49th out of 50 on customer satisfaction (only JJB lower) and PC World joint 43rd alongside MFI and Woolworths.

Low customer satisfaction = low profits
Not surprisingly, DSG’s share price has plummeted in recent months as they continue to lose customers and miss their financial forecasts. Most short-sighted analysts look no further than internet competition for the explanation. But it is possible to sell electrical goods from shops as the following quote from January’s Retail Bulletin confirms:
“One of the most telling things over the Christmas period was that while DSG was complaining about a lack of customers for electrical goods it was a very different story in the electricals departments of John Lewis, which were rammed with people looking to splash out on the latest gizmos.”
And we all know which retailer has the highest customer satisfaction. Maybe DSG is getting the message. Their new CEO, John Browett, was Operations Director at customer-focused Tesco. Maybe poaching a John Lewis executive would have been even better, but they also have very high employee satisfaction at John Lewis!

References
1. www.nationwide.co.uk/consumer_confidence
2. Financial Times / King Sturge, 5th January 2008
3. www.ukcsi.com

Classic charts...and beer

Interesting article in the Economist looking at three of "history's best" charts.

Ever since Tufte published the Visual Display of Quantitative Information*, Charles Minard's graphic of Napoleon's march on Moscow has been the standard example of a thoughtful and compelling "infographic". I even use it in my course...despite its somewhat tangential relationship to Customer Satisfaction Measurement!

Playfair is a fair choice too. He invented, for better or worse, many of the graphic forms that we are familiar with today, including the bar chart, line chart and even the dreaded pie chart. The Economist shows an early attempt to make political mileage out of charts.

The third person featured is Florence Nightingale, who is sometimes thought to have invented the pie chart, but didn't. She did come up with the "Nightingale Rose", or polar area chart, which is the one covered by the Economist. Frankly this is an odd choice, as it is not one of history's best charts by any means. Nonetheless the outcome of the analysis was of great importance, forcing a review of the sanitary conditions of army barracks and hospitals at a time when disease killed far more soldiers than enemy action.

For similar reasons, my preferred third choice would have been John Snow's map of cholera deaths in Soho.

A map that forced the closure of a lethally infected water pump (the Broad Street pump) and finally began to convince people that cholera was water-borne and not spread by smell.

Next time you're in Soho or Carnaby Street, find your way to Broadwick Street, as Broad Street is now known, and have a pint in the John Snow pub. It seems an ideal way to commemorate the closing of an era in which drinking beer was safer than drinking water.

* You can read our review of VDQI here[PDF]

Learning from success?

There's an interesting article in the New Scientist this week critiquing a forthcoming book for making recommendations based on analysis of a few very successful individuals. As the article points out, this approach is common in self-help books and also in business books:

Gates is not alone in believing that society can be improved by studying successful folk. Some of the best-selling non-fiction books of recent years include The 7 Habits of Highly Effective People (over 15 million sold) and Built to Last: Successful habits of visionary companies (almost five years on Business Week's best-seller list). So what do these books tell us about the roots of success? From a scientific point of view, almost nothing.

Ouch. Why not? Problems include a lack of suitable controls (i.e. as well as looking at what successful people have been doing we should look at what the unsuccessful have been doing) and the rarity of huge success compared to normal performance.

In addition, business books often tend to look for very simple, monocausal, explanations. The world is rarely so simple. Even if it is, there's a limit to how successful you can be by aping others—you need to be different from competitors.

You also need to know what to copy. Contrary to the advertising, wearing the same shoes as Kevin Pietersen, or silly earrings, won't make me a great cricketer. Likewise using the same CRM software as Wal-Mart is unlikely to allow you to take over the world.

These and other failings are pointed out in an interesting book, The Halo Effect, which I'll be reviewing here early in the new year. The message is: be very careful about which aspects of top organisations you seek to emulate. They may not be as good as you think they are, and the reasons for their success may not be as simple as they seem.

Better to focus on what your customers say they want, and deliver that consistently, than to get carried away by the latest exciting idea topping the bestseller lists.

Another paradox?

Sean posted an interesting comment in response to my post about Simpson's paradox. His question was

How about another research paradox.You improve in all requirements that are important to customers, significantly in the priorities for improvement, and as expected the satisfaction index increases.However when we look at how the customer rates their overall experience this has not changed.

So the conundrum is: how can satisfaction scores go up for a list of factors that customers say are important to them, but stay the same for an overall satisfaction question?

The first thing to say is that this is very, very, unusual. If you look at the relationship between overall satisfaction and Satisfaction Index there is typically a very high correlation between the two (in the region of .8-.9).

Let'€™s put together a list of potential explanations, and see which ones look most likely.

The way overall perceptions are reported

Overall questions are often reported differently to individual satisfaction scores and the Satisfaction Index. If this question is reported as a percentage "top box" score (and/or bottom box) then it will behave differently to an average, making it more volatile but also less sensitive to some types of change (a top box score would be blind to the elimination of poor performance, for example). It's measuring different things.

As an experiment I looked at the average overall satisfaction score, average Satisfaction Index and % top box on overall satisfaction by month for one client with a very large set of tracking data (42 months). The correlations were:

• Average overall satisfaction with average Satisfaction Index: .86
• Top Box overall satisfaction with average Satisfaction Index: .07

In other words average overall satisfaction is very strongly correlated with Satisfaction Index, but % Top Box is very poorly correlated.

Try looking at the average overall satisfaction score as well as a top box figure to make sure you're getting a fully rounded view.

Sampling error

If you're talking about the change between one survey and another, rather than a consistent trend over time, the issue may be sampling error. Whenever you conduct a survey there is a certain amount of sampling error reflecting the fact that you only talk to a subset of the people whose views you're interested in. This gives every piece of data a margin of error, which will be relatively larger for a single question like overall satisfaction than it is for a composite score like the Satisfaction Index.

Sampling error may be hiding real gains in overall satisfaction by making the score look higher than it should in the first survey and lower than it should in the second.The Satisfaction Index is more resistant to sampling error.

Asymmetric impact

Nigel talks about this quite a lot in his new book. If improvement has mainly been in areas that are "satisfaction maintainers" then you wouldn't expect to see much change in average overall satisfaction, since the shape of the relationship between a given and overall satisfaction is a bit like this:

What you would expect to see is a reduction in the number of customers who are very dissatisfied overall, as the key with givens is to keep performance above a threshold "tipping point".

So is this the answer? Perhaps partially, but it's unlikely that all the priorities for improvement are givens. (It'd be nice to know the impact correlations!)

Improvements to satisfaction maintainers may not result in big changes in average overall satisfaction.

The nature of overall satisfaction

Another possibility is that customers have noticed improvements in specific areas, but this has not yet had time to feed through to their overall feelings about the organisation. If so, we'd expect to see overall satisfaction start to trend up (along with loyalty) after enough time has elapsed. This could be confirmed with statistical models given enough data.

A related explanation is that overall satisfaction is a tricky beast. One of the reasons that the Satisfaction Index is often preferable to a single overall satisfaction question is that it is rooted in specific attributes. Overall satisfaction is a much more nebulous measure, and is bound to incorporate many other influences such as brand image and reputation.

This helps explain why there may be a time lag between improvement in individual attributes and changes to overall satisfaction. It takes much longer for customers to feel generally warmer about an organisation than it does for them to notice specific improvements.

Conclusions

So what is the most likely explanation for Sean's paradox? My guess is that it is probably down to looking at improvements in average scores against a relatively static % top box on overall satisfaction.

If I'm wrong, and Sean's talking about an average overall satisfaction score, then my guess would be a mixture of some of the other issues I've outlined in this post. With a bit more digging around in the data we might be able to eliminate some of them and narrow down the search!