Customer Satisfaction Measurement - The Leadership Factor

Satisfaction measurement - should you go deep or broad?

At the WCCF conference this week there was quite a lot of talk about the advantages and disadvantages of various ways of measuring the customer experience, which I'll probably post more about next week.

One of the things that struck me about what some organisations are doing, particularly with NPS, is that the focus is incredibly strong on getting very frequent data at the lowest possible level (even down to individual call centre agents).

This kind of granularity can be a very powerful tool for change, as it gives people a sense of ownership and accountability for the score, and of course could be used with any metric—there's nothing special about NPS in that regard.

The flipside is that NPS, or to be fair any survey that you can realistically run at these frequencies and volumes, tends to give you very shallow data. You get a score, and perhaps a verbatim comment, but no quantitative insight or specific qualitative information into the reasons behind it.

 

The choice here, it seems to me, is whether you should go for deep or broad understanding of the customer experience. For many organisations the ideal answer will be a combination of both, with some sort of automated (web, email or IVR) option to take the “pulse” of the organisation at a granular level combined with a detailed telephone interview survey backed by thorough analysis and insight.

A ten step guide to getting high satisfaction scores

People often act as if the aim of a customer satisfaction survey is to get a high score, or hit a target, rather than to improve how customers feel about the organisation. Here is our 10 step guide to faking a high satisfaction score without the hard work of satisfying customers.

1. Ask customers about things you know you do well
In order to measure satisfaction properly you have to use the “lens of the customer” rather than the “lens of the organisation”. This partly means language, partly means customers not caring about your internal hierarchies and processes, but most importantly means which aspects of the relationship you measure.

If you come up with your own questionnaire, without involving customers, then by definition you will be asking customers about the areas you focus on. These are likely to be things you are good at, but they may not be the most important things to customers.

The [Building Society] has sent her a customer satisfaction questionnaire. Does she like the decor of her local branch? How long does she have to wait in line? Do staff smile? She has responded positively to each question. There is no opportunity to do anything else.

Suddenly she writes at the bottom, “But I hate you.”
Geoff Ryman – 253, a novel for the Internet

2. Use a positively weighted scale
After all, we know that customers are mostly satisfied, right? So we must need more positive response options. Let's go for the old “Excellent, Good, Fair, Poor” and count everyone who doesn't say “poor” as satisfied. Organisations focusing on their customers rather than the score might make the mistake of opting for a long enough balanced scale to pick apart the scores of satisfied customers, but that's dangerous. What if they all score us 6 out of 10?

3. Force customers into a score you can misinterpret
Well apparently the board don't like that scale. Someone seems to have read a book that said it was “biased”. Oh well, at least if we go for a short enough verbal scale we know the vast majority will give us a “courtesy 4”. Never mind whether that would be a 6 or 9 on a ten point scale, we can still count them as satisfied, and that's what makes the press release.

4. Pretend that “satisfied” is a good thing
“Satisfied” is such a useful word. On the one hand it sounds pleasingly positive—“90% of customers are satisfied”, excellent. And yet at the level of the individual customer, it's so easy to make people satisfied—all you have to do is meet their basic needs and expectations. After all 6 out of 10 is satisfied.

5. Only talk to happy customers
It's much more fun to talk to the customers who like us, why don't we ask them to do the survey for us? They know us much better, after all. Anyway, you shouldn't really worry too much about what Bill says—he's always complaining. Can we delete that score? Excellent.

I think the actions of almost all marketers say, “we’d rather you were happy, but if you can’t be happy, please go away.”
Seth Godin

6. Find an easy benchmark to compare yourself against
This one is 100% bullet proof. No one in your organisation will ever question that your market is completely different from every other market (and yes, you do have the most demanding customers). That means that no benchmark from outside your market is fair, particularly if you don't emerge well from the comparison. The really clever bit is that you can define your market exactly how you want to—genius!

7. Use mystery shopping or internal KPIs as measures of satisfaction
This is like an extreme form of technique number 1. By measuring yourself not only on the things you think are important, but using your own criteria, you can almost guarantee a good result. Companies that were not just focused on getting a high score might use these tools in addition to a measure of customer attitudes, rather than in place of it.

8. Pick your timing
Just made a big change? Don't survey.
Something went wrong? Don't survey.
Rainy day? Don't survey.
England just won the World Cup? Fill your boots.

9. Make sure they know you're watching
Offer to help customers with anything they don't understand on the questionnaire. Stand over their shoulders as they fill it in with the pen you've lent them. Why not offer to fill it in on their behalf, after all it is a bit of a hassle isn't it? Make sure you understand any reasons for dissatisfaction by challenging any low scores, preferably in a slightly aggressive way.

10. Bribery
Offer customers an incentive for filling out the survey—if it's good enough the warm glow should carry over to the scores they give. It's a shame you can't make the incentive conditional on good scores, really (though see point 9).

All of these things are much easier than the prospect of, gasp, actually improving the products or services we offer customers. You may feel that I've ladled the sarcasm on rather thickly here, but you come across these mistakes on a regular basis in current satisfaction measurement. Many organisations take the path of least resistance and adopt the comforting appearance of a high score they don't deserve.

Companies that do this are living in a fool's paradise, and they'll be rudely shaken out of it as initiatives like the UKCSI increasingly provide a published, objective, and stringent measure of customer satisfaction. If you're serious about measuring and improving customer satisfaction you need a tough measure. Find out more in our Improving Customer Satisfaction course.

10 tips for presenting data that will drive action

I take both a personal and a professional interest in presentation. Getting a presentation right means your audience will leave with the right message, understand what the data is telling them, understand what they should be doing, and be inspired to take action.

Presentation is key, and the ultimate test of success or failure is not a ripple of polite applause and a kind word from the Chief Exec, but whether your business changes as a result.

I believe that the main factor which determines the success of a presentation is the clarity of the message, both on individual slides and in the presentation as a whole. This is doubly true of numerical data, such as customer survey results, where it is easy to get bogged down in slide after slide of analysis.

When it comes to presenting numerical data, here are 10 key principles that I have stolen or evolved over the years. These form the basis for most of the recommendations I make on my course.

1. Show me the numbers
Simplicity comes from clarity of message, not dumbing down. If you hide the numbers then I have to trust you, and frankly I don't. By all means use a R2G colour scheme to make the message clearer, but numbers are not innately scary and it's important that they are there for those that want them.

2. Order by something
There is almost always an order more interesting than simply listing items alphabetically or in the way they were presented on the questionnaire. Think about the story the data is trying to tell and then sort it in a way that supports that (e.g. largest values first, largest changes first, largest differences from average first etc).

3. Let the data breathe
Get hold of the Visual Display of Quantitative Information, or something similar, and have your eyes opened to the amount of “chartjunk” and wasted ink in the average table or chart. Fade chart mechanics such as grid lines and axes into the background by changing them from black to grey. You will soon find that you're obsessing about allowing the data room to stand on its own, rather than decorating it. “If the numbers are boring then you've got the wrong numbers.”

4. Tell the truth
This might seem obvious, but it is easy to fool yourself as well as your audience, particularly when evidence can be made to conform to your preconceptions. Make sure you are asking yourself tough questions, and that you can back up your assertions with evidence. I recommend the Halo Effect as an effective antidote.

5. If it's boring, ditch it
When we do a piece of analysis, or ask a question in a survey, it is often because we're wondering if something interesting will emerge. Often it doesn't. If so, I recommend you leave that question or analysis on the cutting-room floor—padding out your presentation with analysis that has no implications for the future dilutes the message and makes you look dull. Have the courage to make decisions about which analysis is meaningful.

6. You are responsible for the actions that are taken
This is a scary one. Using the wrong piece of analysis, or using it in an inappropriate way, can lead to bad decisions. Quadrant charts are notorious in this regard. This can lead to real, damaging, effects in the way your organisation performs. Be careful, and make sure you are confident that you have got everything right. If you want people to take your analysis seriously, you have to take it seriously too.

7. Don't create work for your audience
There is no need to dumb down for your audience, but there is also no reason to be obstructive. Avoid abbreviations, jargon and other barriers. Subtler sins include using poor typography, such as words rotated by 90° in order to fit. This can almost always be avoided, and doing so is a sign of respect for your audience.

8. Analysis and reporting are separate functions
This links in with point 5. Not everything that happens at the analysis stage should end up in the reporting/presenting phase. This is upsetting, since you may want to demonstrate how much hard work you did, but crucial if you want to leave a clear message.

You may also choose to present messages in a different way, or using different charts, than the tools you used to discover them in the first place. A factor score coefficient matrix is unlikely to mean much to the board, but the message about the existence of common factors underlying the items on the questionnaire is perfectly comprehensible.

9. Delete first, then add
Perhaps the easiest way to achieve number 3 is to remove the unnecessary ink that Excel and other programmes tend to add to graphs. Clearing away the clutter, then fading the chart mechanics into the background, will free up lots of empty space. This is a good thing in itself, but it also allows you the room to add elements which help the message stand out. Don't be afraid to annotate directly onto your charts where it helps focus attention on the message you're trying to tell.

10. Find a message, then sell it
The single most important thing is that you know what the analysis is trying to say. A good discipline when preparing presentations is to write “headline style” titles. By using strong declarative statements (“Satisfaction is up in the North West”) rather than category headings (“Regional satisfaction vs. 2008”) you are forced to think about what the message is, as well as making a much more engaging presentation. If you can't find a headline, then see point 5.

Looking across these headlines should allow you to see a building story that will leave the audience in no doubt about what your main message(s) are. If not you need to decide what they are and restructure the presentation to deliver them clearly.

How scales work

Bear with me, this will get relevant!

The world of climbing has, somewhat oddly, been riven with controversy over scales. One of the UK's top climbers, James Pearson, made a first ascent of a very hard and dangerous new route which he named “Walk of Life”. So far, so good.

He decided to give the route a grade of E12 which, without getting into the technicalities of the British trad grading system, would make it the hardest climb in the world. Many armchair critics were sceptical about the grade, and the situation got even more heated when Dave MacLeod repeated the route and gave it a grade of E9 (still very hard and dangerous, mind you).

 

[Image source: http://davemacleod.blogspot.com/]

The poorly-informed and ill-tempered debate that followed in the climbing media (which both Dave and James rose above) focused on whether the grading scale is “broken”, or whether climbers give grades in order to attract publicity/make themselves look hard.

I think this is missing the point completely. The real lesson is that it is impossible for humans to give a completely objective measurement of anything. This is similar to the objection that “one person's 8 is another's 9” or, for that matter, “one person's satisfied is another's very satisfied”.

It's never a good idea to rely too much on a single score given by a single person, but the law of large numbers means that, over a sufficient number of people, average scores will stabilise and be reliable. This is the magic of quantitative surveying—averaging lots of unreliable scores gives us a reliable average.

Interestingly, the few sensible comments about the Wall of Life grading debate draw attention to the fact that climbing grades are always established by consensus. This is simply an informal version of the same phenomenon.

Analysing qualitative data

Delegates on my training course often ask me about analysing qualitative data, usually in the form of verbatim comments from customers. Compared to quantitative data, qualitative data often seem bulky and time-consuming—surely there must be an easier way?

Sadly there isn't, really. There is no software with a magic “code all this data” button, and even if there were it would be a drastically bad idea to use it. This is an area in which a human brain needs to be employed for any real insights to emerge. Mind you, so is quantitative analysis...but that's a post for another day!

There is software designed to make your life easier when it comes to analysing qualitative data. Collectively these programmes are known by the stunningly unwieldy acronym CAQDAS*. If you want a full list you can find one at the ASC website (caution—ugliest website ever). Prominent names include NVivo, Qualrus and The Ethnograph.

While they can't do the analysis for you, computer programmes do give you tools that make your life easier. You can get your computer to produce a word frequency count and concordance (key words in context), as well as to search for key phrases. But this kind of automated work is rarely very interesting.

CAQDAS software really comes into its own in easing the process of building up a coding frame and then using that to generate theory. Many CAQDAS programmes are built around an academic approach known as Grounded Theory, which is great for inductive reasoning in your PhD thesis but, frankly, probably overkill for your customer satisfaction survey.

Whatever software tools you use, effective analysis of qualitative data will always take time and care. Coding is a back-and-forth process that takes frustratingly longer than you expect. Above all you need to remember that coding can take you away from the depth of the underlying data—the codes are there to help you think, not to convert your data into arbitrary quantifiable categories!

* Computer Assisted Qualitative Data Analysis Software or sometimes QDAS or QDA for short!

Zen and the art of customer satisfaction measurement

It's tempting to over complicate the things that are important to you. This means that when we sit down with clients to design a satisfaction survey (important to us) to measure the feelings of their customers (important to them) there's often more pressure to add  than to subtract.

This comes from a laudable desire to understand customers in great detail, but risks alienating customers with the very tool that is supposed to be getting to the bottom of their feelings.

That's why we go on and on about the importance of exploratory research. A customer survey must be based on questions which are relevant to customers, and reflect the things which they're thinking about when they make their judgements about you. Frequently these things are painted with a broad brush, but that's how customer attitudes work. No matter how much we would like them to be precise and actionable, customer attitudes will always be impressionist rather than photorealistic.

The zen art of customer surveys is to accept this occasional imprecision as the price for an accurate measure, and to find ways round it from the point of view of actionability. Our main weapon here is collecting verbatim comments to illustrate the individual stories behind each low score. This balances the broad-brush measurement, that reflects the way customers see the world, with the precision and actionability an organisation needs to improve. Over-complicating the list of requirements to be scored only frustrates the customer and spreads a single impression across five apparently different areas.

Another day, another nonsense survey

According to a survey, 95% of journalists don't care about truth.

Bold claim? Not really, because "according to a survey" is a get out of jail free card that (I'm fairly sure) prevents anyone from being able to sue me.

The sad truth is that no-one really believes what surveys reported in the press say, unless they happen to reinforce their own pre-existing beliefs. Why? Because the vast majority of surveys enthusiastically reported in the press are horribly misused.

Ben Goldacre of Bad Science is one of many bloggers to pick up on the BBC's monumentally stupid reporting of a survey about the happiest places in the country. The researchers made it perfectly clear that none of the differences are statistically significant. In simple terms that means that the differences they found may reflect random survey error rather than real differences. In other words, there is no story. The message here is "some places may or may not be happier than others. We don't know." Understandably, the BBC didn't think that was especially newsworthy.

Instead of reporting the real message, they decided to ignore the stuffy old statistics and just present the apparent differences as fact. This is, frankly, lying. When called on the issue, their response was to say that it's okay, because the piece is of a "light-hearted nature".

The problem is that every time this happens it erodes the basis on which survey research, and statistics more generally, rest. Statistical significance is not something we can ignore when it suits us, it is fundamental to knowing how much faith we can put in a statistic.

Events such as this are a powerful argument for replacing statistical significance, which can look stuffy and irrelevant, with confidence intervals. Going back to my survey at the beginning of this post, it won't surprise you to learn that this isn't a statistically significant finding. With a confidence interval of +/- 30% the real answer could be as low as 65%. But I doubt it.

We blogged on the difference between significance testing and confidence intervals back in March.

Offshoring customer service - "good profits" and "bad profits"

Fred Reichheld has a gift sometimes for coining a neat turn of phrase. One I really like is his concept that there are "good profits" and "bad profits".

Good profits come from a satisfied and loyal customer base that has few problems and enjoys doing business with you. Bad profits are usually about cutting costs—they come from measures that increase profits in the short term but hurt customer satisfaction and loyalty. in the long term bad profits can lead to a vicious cycle of falling margins, plummeting loyalty and ever greater commoditisation. Bad profits are bad both for organisations and for their customers.

Offshoring customer servce seems like a classic case of bad profits. Customers don't like it, and organisastions know they don't like it, but offshoring still goes on because companies are desparate to save costs and benefit from the easy boost to profits.

Now we have some research into the long term effects. Three american academics (including the brains behind the ACSI) have published a paper summarised in a Wall Street Journal article that shows offshoring customer service damages customer satisfaction and loyalty. Historically, such a drop in ACSI goes along with a fall in market capitalisation. The paper also shows that, by contrast to offshoring customer-facing functions, back-end systems can be safely offsourced as long as efficiency is maintained.

The moral of the story? If you're interested in long-term success give customers what they say they want. Doing best what matters most to customers over the long term will lead to financial rewards, and helps to prevent the race to the bottom of commoditisation and cost-cutting. But offshoring in itself is not the problem, the problem is that customers don't like specific functions being offshored.

Brand tags

Every so often the blogosphere comes up with a good idea. Once in a blue moon it's a really good idea.

Brand image has always been a tricky beast to pin down, but most people would define it as something like the sum of all the attitudes and associations that consumers hold about a particular brand. Some different ways of putting this:

It has always seemed to me that your brand is formed primarily, not by what your company says about itself, but what the company does. Jeff Bezos, CEO, Amazon.com

Consumers build an image [of a brand] as birds build nests. From the scraps and straws they chance upon. Jeremy Bullmore

A brand is a living entity—and it is enriched or undermined cumulatively over time, the product of a thousand small gestures. Michael Eisner

We have the power to shape brands to be what we want… Wally Olins

A brand image is shaped by the products and services themselves, by the consumption environment, by marketing communications and, perhaps most importantly, by the behaviour of the organisation and its employees.

So how do you measure something so vast and intangible? One technique is to ask people what associations they have for a particular brand, something along the lines of "what 3 words come to mind", and over a sample of hundreds some interesting patterns will emerge.

Brand tags is a website that does exactly the same thing, across a whole host of brands, and represents the results in the form of a "tag cloud" similar to the one to the right of this blog. A fascinating concept, and there are some revealing results. You can take part in the tagging or simply browse through the  brands.

Here's a snippet of the cloud for a famous sports shoe brand:

Can you guess who it is? If not peek at the image title/alt tag, or listen to more hip hop!

Public sector customers?

I received an interesting email today from the Head of Customer Development at a large local authority. He asked whether the different relationship that public sector suppliers have with "customers" should dictate any differences in approach.

My answer was that, although the different relationship entails some surface changes to approach, the core remains very much the same. It means, for example, that there is little point in asking conventional loyalty questions like "will you still be a customer" or even the ubiquitous "would you recommend", but customer satisfaction is still very relevant.

The starting point, whatever your organisation, should be to think about what your desired outcomes are. In the private sector, the company may want increased customer retention, higher sales, larger market share, higher profits and so on.

What are the equivalent desired outcomes for the public sector? Once those have been pinned down it will be much easier to see how to approach both service delivery and customer research. For instance you might set goals based on reducing complaints, or increasing satisfaction as an end in itself, rather than as a step on the road to greater profits.

What do you think—are public sector "customers" fundamentally different from private sector customers? Should public sector organisations manage their customer relationships differently?