A site I keep coming back to is Peter Norvig's* excellent piece on warning signs in the design and interpretation of experiments and the statistics that back them up. This is well worth reading and absorbing.
Even if you're already familiar with all the traps he outlines, it's rare to see them explained so concisely and so clearly all in one place. Many of them are relevant to research in general, and customer satisfaction research in particular.
I especially like Norvig's very clear example of the care with which probabilities about tests have to be used to draw conclusions about the probability of an event ("Warning Sign 14"). Let's make it relevant to customer research. Imagine that we are a credit card issuer, and we know that 80% of customers who defect have not used their card for at least a month. By contrast only 10.5% of customers who are not intending to defect will go a whole month without using their card. All we have to do is set up a trigger so that we call any customer who has been inactive for a month and we should be able to significantly decrease defections.
So what are the chances that an inactive customer will defect? Assuming that the base defection rate is 5%, the chances are actually less than 30% that each customer we ring is about to defect. Why so low? because defection is a comparatively rare phenomenon—even with a very accurate test the true defectors are outweighed by false positives. Exactly the same is true, as Norvig shows, for medical screening tests for breast cancer etc. The logic is explained in the table below:
So the accuracy of the test in correctly identifying defectors and non-defectors (technically the sensitivity and specificity) is very good (80% and 89.5% respectively), but this does not mean that someone who is inactive has an 80% chance of defecting.
Now go and read the rest of the article!
* Peter Norvig is a computer scientist who works for Google. His articles on a range of subjects are worth reading if you're interested in clear thinking and lucid, witty, writing. His online resume gives you an idea of his background, and also of how good an employer Google is:
"Note to recruiters: Please don't offer me a job. I already have the best job in the world at the best company in the world."