Customer Satisfaction Measurement - The Leadership Factor

Truth is another story

This post in a nutshell: stories are powerful, but dangerous.

Powerful, because they are the best way to persuade people to do what we want them to do (journalists, lawyers and salespeople know this well). Dangerous, because we also use them to sell easy explanations to ourselves.

 

Fooled by ourselves

In The Black Swan, Nassim Nicholas Taleb talks about the "Narrative Fallacy"—our tendency to believe stories we've made up to explain facts. Pretty much every newspaper story about scientific research is guilty of this (see for example this post by Kaiser Fung).

The problem is that we tend to be happy with an explanation which is plausible, as long as it is consistent with the evidence (particularly if it fits with our preconceptions), whilst forgetting that there are lots of other potential explanations. By jumping to conclusions we automatically cut off any further investigation of the real explanation, so this process tends to be self-fulfilling.

A key part of the scientific method is designed to tackle exactly this. Proper testing is all about thinking up ways our story could be shown to be wrong, not piling up more evidence that it is right. Sadly the testing part of the process rarely transfers into business.

Science: Facts->Hypothesis->Testing

   vs.

Business strategy: Facts->Hypothesis....... 

So what happened to fact-based management? It has been a victim of three fatal weaknesses:
1) Proper scientific testing of causality is difficult, sometimes impossible.

2) Science is a bit nerdy, and the people who run businesses were more often the captain of the school rugby team than they were the chess club.

3) Business likes to perpetuate a myth of itself as slightly macho, seat of the pants, instinctive and passionate. This is all very well if you want an organisation staffed entirely with Apprentice contestants (shudder), but doesn't help if you want to make good decisions.

Proper science is the only way to find out what actually works. As I've said before on this blog, why not put some of your assumptions to the test?

 

The responsibilities of the story-teller

If we have a duty to ourselves, and to our businesses, not to be fooled by our own glib explanations and preconceptions, we should also do our best to make sure that we don't lead others down the garden path.

It may not always feel like it, but our stories can be just as compelling for other people as they are for us. We need to be very confident what we're saying is true. The cynic in me thinks that many people would be paralysed with terror if they thought there was the slightest chance that what they were saying might actually change the way anything was done...but shouldn't we at least act as if it might?

Stories are, without doubt, the best way to make people take notice of what you're saying. But before you rush off to create your next story, make sure it's grounded in evidence. You don't necessarily need to show your working, but you do need to have done it. Great writers spend most of their time researching and rewriting—shouldn't you do the same?

Surveys - we need facts to make change

So part 2 in my series of posts about improving customer satisfaction, using information from your satisfaction surveys.

Last time we looked at three general lessons:

• There is no average customer.
• Too much and too little are not equally bad.
• You can't talk to customers too much (unless you're selling).

Now I'd like to turn to the question of how you can make best use of a customer survey to inform change. Bruce Temkin has a really interesting blog post in which he argues that research agencies need to be supplying more "contextual insight" and less "statistical analysis" if they want to help their clients. I think this is 100% right.

The interesting thing is that many researchers turn to a statistical fix when they want insight, which is like buying a CRM system to get closer to customers—it might help, but it doesn't address the fundamental need. The challenge is to identify the fundamental questions we want to answer with the research.

A survey can basically give you three types of information:

• Attitudes—how customers believe they feel/felt or think/thought
• Behaviours—customer perceptions of what you or they did
• Classification—what you know, or customers are happy to tell you, about who they are

Surveys are a great tool, but they can't do everything. For one thing, they tell you less about performance than about customer perceptions. Often those are closely related, but they're rarely identical. In other words, if you get low scores for delivery times it doesn't necessarily mean that your delivery performance is poor. When it comes to improving, managing customer expectations can be much easier and more effective than trying to take slices off your time. Perception is reality from the customers point of view, but that doesn't mean it maps directly to what you are doing.

How can the survey help? By moving beyond attitudes (like satisfaction scores) to the behaviours and processes that are driving them. We can start to ask customers how long their last delivery took, and how long they expected it to take. Now we can start to match that data up to our internal measures to reach evidence-based conclusions (rather than speculation) about what is causing the low score. Customers might be dissatisfied because:

• Our performance is bad—we need to speed up our deliveries. But let's not dilute our effort by working across the board. Can we identify particular products or customer types where delivery time is more of a priority? Where are the "cliff edges" in the data—often there are points of tolerance with issues like this where satisfaction suddenly drops off.
• Customers think our performance is worse than it is—asymmetric impact means that they remember the missed deliveries more than the on time ones. Communicating real performance and celebrating success can help, but make sure you don't cheat. The train companies have shot themselves in the foot by using a bizarre definition of "on time" when they communicate the percentage of departures on time.
• Customers' expectations are unreasonable—they want delivery in a time that defies the laws of physics. Communication and controlling expectations is the answer here (although Seth would probably tell you to change the laws!)

What else could we do to improve? Ask customers whether or not they used the order tracking facility on your website. If the ones who did are more satisfied than the rest then push it harder, otherwise try something else.

The lesson is that survey data does not begin and end with attitudes, even if that's what we're mainly concerned with. Surveys are not a good measure of performance, and to make them useful in improving performance you have to use them intelligently to inform what you do. Find out, empirically, what is causing customers to feel the way they do, and you can start to plan concrete changes that will result in visible gains in performance.

Never trust an average

There aren’t many statistician jokes, I imagine, but here’s the only one I know.

Three statisticians go out on a hunting trip. After a while they see a deer. The first shoots and misses to the left. The second misses to the right. The third shouts “we got him!”

The point? Averages, even when true, are a mathematical fiction rather than a concrete reality—the notion of 2.4 children as the average nuclear family illustrates this well. We should bear this in mind whenever we use numbers to summarise people or experiences. Let me give you two examples.

Customers never have an average experience

Twice recently, when talking about excellent companies with a strong (and deserved) reputation for customer satisfaction, I’ve had the wind taken out of my sails by people objecting “well I think they’re rubbish!”, and they go on to explain exactly why. The thing is we’re both right, but I’m talking about an excellent average, and perhaps one or two even better highlights from conferences and case studies, while the other person is talking about their own unique experience. Even the very best companies get it wrong sometimes.

The lesson here is that, even if you are very good, it’s worth addressing the occasions when it goes wrong. In particular it’s a good idea to keep working hard on making sure you are as close to flawless as possible on the “givens”. Tracking percentages of low scores, as well as averages, is a good way to do this.

I am not a number

Where averages are really dangerous is when they are used to draw conclusions about groups of people (here we are a whisker away from statistically-backed bigotry if we’re not careful). The problem, really, is that statisticians know what they mean when they say “men are better than women at X” or “girls are better than boys at Y”. They mean, usually, that there is a small but statistically significant difference in the averages for the two groups. The problem is that’s not how normal people interpret the statement—which, at best, would be that almost all girls are better than almost all boys at Y.

Let’s look at an example I cooked up. Say we have administered a test to 1,000 girls and 1,000 boys, and they all have a score.

Gender	Mean	N	SD	Confidence Interval
Male	7.0	1000	1.4	+/- 0.1
Female	7.6	1000	1.5	+/- 0.1
Total	7.3	2000	1.5	+/- 0.1

The analysis shows that girls have a higher average score than boys, and we can run a t-test that shows this is a highly statistically significant difference. Confidence intervals tell us the same thing. In other words girls are better than boys at this. But what if we get away from the averages and take a look at the distributions underlying them, i.e. get back to the raw scores?

I bet you weren’t expecting that much overlap, were you? About 35% of females have a score lower than the average for males, and about the same number of males have a score higher than the average for females.

The lesson is that comparing averages, though it can be useful, tells us very little about any particular individual. Next time you see a story of this pattern in the paper (or in books that make a fortune using tiny laboratory differences to back up cliches about empathy and reading maps), ask yourself what the underlying distribution looks like.

The Decisive Moment

“X is like Y, because...” is a useful way to generate fresh thinking about pretty much any topic (and quite a good qualitative research technique to boot).

“Presentations are like films, because they are about telling a compelling story through a primarily visual medium” is so well established that it is now a cliche. Cliff Atkinson even suggests using a three act structure for your presentations (actually not as silly an idea as it first sounds, albeit over-prescriptive).

The lean forward moment

I've just finished a great book on film-making and editing called “The lean forward moment”, which has lots of useful advice about telling stories.

In a nutshell Hollyn argues that stories are about change, and that change (of any kind) can be used to signal important turning points to the audience. This might mean a change of acting style, a physical movement, a new camera shot, different lighting, music starting (or stopping) and so on. It doesn't matter what changes, as long as the moment has a chance to register.

As Hollyn points out, this is precisely the technique used by many public speakers when they use pauses to allow key points to land. They might also use visuals, props, changes in the pitch or rhythm of their voice, and so on. Change grabs attention.

Moments of truth

I, like Cartier-Bresson, believe that most things in life have decisive moments—lean forward moments—which are particularly powerful. When talking about the customer experience we call these “moments of truth”. Can we apply Hollyn's techniques to get the most out of moments of truth with customers?

If we take each customer interaction as a film scene, Hollyn suggests that we approach the scene with three questions:

1. Whose scene is this? Many organisations act as if it's their scene, but it should be the customer's.
2. How does that character change from the beginning of the scene to its end? Clearly we'd like them to be delighted by the way we've surpassed their expectations, not disappointed.
3. Where, exactly, does that change occur in the scene? Ah, now that's an interesting question. What are the moments of truth? Once we realise that this scene belongs to the customer we can start to look at the turning points inside their minds as they go through the change identified at question 2. We can start to control how the scene's going to end by understanding that the outcome is not based on a gradual progression, but pivots around a small number of key moments.

Reporting that motivates change

Last week, Richard and I ran the Stakeholder Showcase in London (which did not cost the £99,999 it's currently advertised at! We're good, but we're not that good).

One of the things I talked about was that the way you report and communicate the survey results has a big impact on whether or not change happens. For me, this is the ultimate test of whether you have succeeded or failed with the survey.

For this reason, I would encourage companies to embrace "gimmicks" when they are an effective way to engage staff with the cutomer survey. Most of the time, the best way to do this is by bringing the survey alive through real customers' stories. Verbatim comments, focus group videos, VoxPops and even cardboard cutouts are all effective techniques.

Once you've won their hearts, you can start worrying about their minds. On my analysing and reporting course (worth every penny of £100,000, but a snip at £325) I talk about ways to present survey data to motivate and enable action. I think there are three essentials:

• Focus. You can't improve everything at once. Keep the number of Priorities for Improvement as low as possible.
• Ownership. Give people a score as specific to them as you can. This sense of ownership (and responsibility) helps concentrate their minds on the importance of the customer. Make sure that PFIs are owned by the people who can affect them. Internal league tables of branch scores are a great way to motivate change.
• Actionable outcomes. People must know not only what to change, but how to change it. Verbatim comments are usually your best source of specific customer stories about where they are being let down.

Words, words, words

A perennial problem for researchers is how to deal with the reams of verbatim customer comments that a useful survey produces. We can sort, group and code; but ultimately the only real way to get to grips with them is to read them.

By and large this is a good thing, but when it comes to presentation it's much less clear what to do. We might select a few "juicy" quotes to illustrate a particular point. We might even play audio clips or use "VoxPops", which are certainly compelling. But there's no clear way to present a visual summary of customer (or employee) comments.

I think we should consider tools like the image above, a "Wordle" of all the posts from this blog, as a way to point the audience towards the benefits of reading the comments for themselves.

Ban Arial (as well as Comic Sans)

Forgive me if this gets a bit ranty—type is something I feel passionate about.

I am becoming increasingly, and perhaps irrationally, irritated by the ubiquity of the typeface Arial in business communication. It's not a bad typeface, nor do I really object to its close association with Microsoft. It does grate on my nerves that it is a poor man's copy of Helvetica. A copy that is close enough to fool most people, while lacking all the understated elegance and subtle beauty of its more thoughtfully designed cousin (Helvetica is on the right, by the way):

No, even that is not enough reason to campaign for a ban of Arial. After all, Spartan (one of my favourite typefaces)is a knock-off of Futura. What I really object to is the utter ubiquity of Arial.

Typefaces are the written equivalent of tone of voice. They can be lyrical or straightforward, softly-spoken or bellowing, playful or severe. Choosing your typeface carefully is an important part of communicating your message...so why is it that every single business presentation is in Arial?*

Arial is safe. Arial is sturdy, functional, corporate, and boring. Really, really, boring. Helvetica is a bit bland, to be honest, but at least it has a certain Swiss elegance. Arial is boring and ugly. It's the type equivalent of a 1960s concrete office building.

We can do so much better. On the web we are hamstrung, dependent on the end user to have the typeface we want to use installed on their machine. No such limitation pertains to slideware, in which we can embed TrueType fonts if we need to share the presentation with others or use it on another machine. Complete creative freedom!

So here is a call to arms—discover your typographic voice. Cut loose from the lumpen fetters of Arial and find a font that reflects your individual message and tone. If you need some inspiration, I recommend browsing through myfonts.com. Jos Buivenga, for example, has a range of truly high quality free fonts, all of them brimming with personality.

 

Isn't that a refreshing change of tone from dull, dull, Arial?

* And why always BOLD...unless you are the sort of pompous dullard that believes talking loudly and monotonously shows how important you are?

10 tips for presenting data that will drive action

I take both a personal and a professional interest in presentation. Getting a presentation right means your audience will leave with the right message, understand what the data is telling them, understand what they should be doing, and be inspired to take action.

Presentation is key, and the ultimate test of success or failure is not a ripple of polite applause and a kind word from the Chief Exec, but whether your business changes as a result.

I believe that the main factor which determines the success of a presentation is the clarity of the message, both on individual slides and in the presentation as a whole. This is doubly true of numerical data, such as customer survey results, where it is easy to get bogged down in slide after slide of analysis.

When it comes to presenting numerical data, here are 10 key principles that I have stolen or evolved over the years. These form the basis for most of the recommendations I make on my course.

1. Show me the numbers
Simplicity comes from clarity of message, not dumbing down. If you hide the numbers then I have to trust you, and frankly I don't. By all means use a R2G colour scheme to make the message clearer, but numbers are not innately scary and it's important that they are there for those that want them.

2. Order by something
There is almost always an order more interesting than simply listing items alphabetically or in the way they were presented on the questionnaire. Think about the story the data is trying to tell and then sort it in a way that supports that (e.g. largest values first, largest changes first, largest differences from average first etc).

3. Let the data breathe
Get hold of the Visual Display of Quantitative Information, or something similar, and have your eyes opened to the amount of “chartjunk” and wasted ink in the average table or chart. Fade chart mechanics such as grid lines and axes into the background by changing them from black to grey. You will soon find that you're obsessing about allowing the data room to stand on its own, rather than decorating it. “If the numbers are boring then you've got the wrong numbers.”

4. Tell the truth
This might seem obvious, but it is easy to fool yourself as well as your audience, particularly when evidence can be made to conform to your preconceptions. Make sure you are asking yourself tough questions, and that you can back up your assertions with evidence. I recommend the Halo Effect as an effective antidote.

5. If it's boring, ditch it
When we do a piece of analysis, or ask a question in a survey, it is often because we're wondering if something interesting will emerge. Often it doesn't. If so, I recommend you leave that question or analysis on the cutting-room floor—padding out your presentation with analysis that has no implications for the future dilutes the message and makes you look dull. Have the courage to make decisions about which analysis is meaningful.

6. You are responsible for the actions that are taken
This is a scary one. Using the wrong piece of analysis, or using it in an inappropriate way, can lead to bad decisions. Quadrant charts are notorious in this regard. This can lead to real, damaging, effects in the way your organisation performs. Be careful, and make sure you are confident that you have got everything right. If you want people to take your analysis seriously, you have to take it seriously too.

7. Don't create work for your audience
There is no need to dumb down for your audience, but there is also no reason to be obstructive. Avoid abbreviations, jargon and other barriers. Subtler sins include using poor typography, such as words rotated by 90° in order to fit. This can almost always be avoided, and doing so is a sign of respect for your audience.

8. Analysis and reporting are separate functions
This links in with point 5. Not everything that happens at the analysis stage should end up in the reporting/presenting phase. This is upsetting, since you may want to demonstrate how much hard work you did, but crucial if you want to leave a clear message.

You may also choose to present messages in a different way, or using different charts, than the tools you used to discover them in the first place. A factor score coefficient matrix is unlikely to mean much to the board, but the message about the existence of common factors underlying the items on the questionnaire is perfectly comprehensible.

9. Delete first, then add
Perhaps the easiest way to achieve number 3 is to remove the unnecessary ink that Excel and other programmes tend to add to graphs. Clearing away the clutter, then fading the chart mechanics into the background, will free up lots of empty space. This is a good thing in itself, but it also allows you the room to add elements which help the message stand out. Don't be afraid to annotate directly onto your charts where it helps focus attention on the message you're trying to tell.

10. Find a message, then sell it
The single most important thing is that you know what the analysis is trying to say. A good discipline when preparing presentations is to write “headline style” titles. By using strong declarative statements (“Satisfaction is up in the North West”) rather than category headings (“Regional satisfaction vs. 2008”) you are forced to think about what the message is, as well as making a much more engaging presentation. If you can't find a headline, then see point 5.

Looking across these headlines should allow you to see a building story that will leave the audience in no doubt about what your main message(s) are. If not you need to decide what they are and restructure the presentation to deliver them clearly.

Choosing colours for charts

As with most aspects of reporting and data presentation, restraint is the key when it comes to choosing colours. You may have noticed that experts in charting, such as Edward Tufte or Stephen Few, tend to choose very muted colours for the majority of their work. Bold colours are used sparingly to highlight and pick out key areas.

In Envisioning Information Tufte quotes Ann Driver to explain way, by analogy to music:

"Music without accent lacks coherence....Conversely, if every note, word or movement is stressed, the result has even less meaning."

Charts in which all the colours are bold are like speakers who shout the whole time, or music without variation. They are boring, and the key information is lost in the general clamour. On top of that, bold colours tend to clash with each other.

Choosing the colours that go is not just a job for Dulux, though, and you don't need a degree in colour theory to make sensible choices for charts. You can't go wrong, for example, with grey as a base colour and any bold colour as a highlight (red is a popular choice).

If you want to get more adventurous, my advice would be to ignore the normal RGB and CMYK colour models, which are hard to think in, and work with Hue, Saturation and Value (HSV). This gives us the very intuitive colour triangle, which I've reproduced here after the version in the excellent Color &Type.

This is a simplification of the real triangle which is a continuum from black to white (left to right), and from fully saturated at the top to desaturated at the bottom. The three highlighted axes are fertile avenues for finding pleasing colour combinations. If we choose red as our basic hue:

That gives us three options: red and grey; dark red and white; light red and black, all of which work well. Choosing a more complex colour scheme, using more than one basic hue, is harder. But the same basic principles of contrast apply—using tints and shades (in other words varying the saturation and value) will often make colours work together. Consider the notoriously difficult red and blue:

Using both colours bold is disastrous. They clash badly, making it almost impossible to look at.

By using either a tint or shade of blue we can achieve a more pleasing result, one that will make it easier to appreciate the data in a chart.

It's worth noting how warm colours such as red have a tendency to rise above cool colours such as blue, which gives the illusion that the red is "floating" above a blue background. Used with care this can be a very valuable technique to highlight key elements of a chart or slide.

Another day, another nonsense survey

According to a survey, 95% of journalists don't care about truth.

Bold claim? Not really, because "according to a survey" is a get out of jail free card that (I'm fairly sure) prevents anyone from being able to sue me.

The sad truth is that no-one really believes what surveys reported in the press say, unless they happen to reinforce their own pre-existing beliefs. Why? Because the vast majority of surveys enthusiastically reported in the press are horribly misused.

Ben Goldacre of Bad Science is one of many bloggers to pick up on the BBC's monumentally stupid reporting of a survey about the happiest places in the country. The researchers made it perfectly clear that none of the differences are statistically significant. In simple terms that means that the differences they found may reflect random survey error rather than real differences. In other words, there is no story. The message here is "some places may or may not be happier than others. We don't know." Understandably, the BBC didn't think that was especially newsworthy.

Instead of reporting the real message, they decided to ignore the stuffy old statistics and just present the apparent differences as fact. This is, frankly, lying. When called on the issue, their response was to say that it's okay, because the piece is of a "light-hearted nature".

The problem is that every time this happens it erodes the basis on which survey research, and statistics more generally, rest. Statistical significance is not something we can ignore when it suits us, it is fundamental to knowing how much faith we can put in a statistic.

Events such as this are a powerful argument for replacing statistical significance, which can look stuffy and irrelevant, with confidence intervals. Going back to my survey at the beginning of this post, it won't surprise you to learn that this isn't a statistically significant finding. With a confidence interval of +/- 30% the real answer could be as low as 65%. But I doubt it.

We blogged on the difference between significance testing and confidence intervals back in March.