Customer Satisfaction Measurement - The Leadership Factor

Involving employees

One of the questions we're often asked is how to get ideas for improving satisfaction. There are many answers [PDF], but one of the easiest (and best) ways is to ask employees for ideas.

Whether it's improving customer satisfaction, employee satisfaction, or process efficiency, employees have a wealth of knowledge and ideas that you could benefit from tapping. This is an area where new technology can really offer something, turning a corporate intranet into a vibrant discussion forum instead of a tightly-controlled propaganda pamphlet. If you trust your employees and value their input.

You could use specialist tools or adapt open source solutions for the job, but either way isn't it time you got your employees talking?

The paradox resolved...but what should we do about it?

Last time I set you a puzzle—how could an organisation improve satisfaction with each of its products, but have lower satisfaction overall? This is one form of a statistical anomaly known as Simpson's paradox which deserves to be better recognised than it is.

Simpson's paradox in worth reading about in detail, but the short version is that overall data shows the opposite effect to that found in subgroups. It happens when there is a hidden weighting variable, which acts as a confounder when we draw conclusions.

The hidden variable in this instance is the change in sample size for each product. The overall score is lower because it is based on more large player customers (who are less satisfied) and fewer tiny player customers.

	2006		2007
	Sample size	Satisfaction	Sample size	Satisfaction
Overall	600	81	600	78
Tiny player	300	85	100	86
Small player	200	78	200	79
Large player	100	73	300	74

We'll assume that this sample is perfectly representative of the population—in other words this company really has sold far more large players this year. If not, then the problem could well be with the sampling, and ensuring representativeness would be the answer

So, is satisfaction going up? In a word…yes and no. How's that for fence-sitting? Interpreting this data is not straightforward. Satisfaction is improving for each product, but we need to think about how to interpret the change in proportion of sales for each one. Our customer base, overall, is less satisfied than last year.

Strategically, this shift towards products where we perform less well could be very significant and worrying. This wouldn't be the first company to slide into oblivion by making great products that no one buys (even the mighty Steve Jobs fell foul of this).

Conclusion

• Be aware that changes to the make up of your sample can affect the overall result
  • Make sure you sample properly
  • Consider weighting to adjust for problems
• Look out for instances of Simpson's paradox
  • Don't interpret relative proportions causally
  • Read about the Berkeley sex bias case for a striking real example
• Think about interpretation when it does happen
  • Performance is improving with each product
  • But is there a deeper problem?

A puzzle

Imagine you are a manufacturer of consumer electronics, specialising in MP3 players. You measure customer satisfaction once a year, and you split the results to show the scores for each of your three MP3 players: a large one, a small one and a tiny one. Customer satisfaction is highest with the tiny player and lowest for the large player.

The good news is that, since last time, satisfaction has risen with all three players. Great...but hang on — the overall satisfaction index for the business has fallen!

What on earth is going on? This situation is one that can easily trip organisations up, particularly when they have significant customer segments, as in this case, or business units. Is this organisation getting better or worse? Your challenge for next time

• Can you explain what on earth is going on?
• What could this organisation have done to avoid this paradox?

Hidden decisions that ru(i)n your business

One of the points I try to make as forcefully as I can in my training course is that seemingly trivial decisions about reporting can have important effects in the real world.

The biggest danger area for many businesses is data flowing into traffic-light coded dashboards, where pre-defined criteria are designed to spot any warning signs and highlight them for attention. How carefully have those criteria been chosen? Should such criteria ever be allowed to take the place of scrutinising the data itself, looking for patterns and trends?

Nicholas Bizzantz has an interesting post about the problems Sachsen LB (a publicly owned German bank) ran into despite—or perhaps because of—a dashboard full of green lights. The key point is:

People who deliver green lights instead of the underlying numbers have made a decision instead of supporting the decision-making process.

Exactly! Someone, somewhere, has made a decision that hid the key data from executives under the guise of simplifying, and the business suffers as a result.

What does a successful dashboard get right? Some essentials:

• Monitor the right things (attitudes like customer satisfaction as well as financials)
• Sort, group and organise for clarity
• Show the numbers
• Consider trends and relationships

Get all those right and your business should avoid nasty surprises.

More on dashboards in this article[PDF] from Stakeholder magazine.

New Customer Satisfaction Book

I've just finished reading "Customer Satisfaction: the customer experience through the customers' eyes". Admittedly, it has been written by some colleagues of mine, but setting that aside, it's a fantastic roundup of what's going on in the customer management world. You can visit the book's website here and I'm going to post again shortly, discussing some of the latest topics that the book covers.