A recent academic paper, The Predictive Ability of Different Customer Feedback Metrics for Retention, is likely to stir things up in the debate about which is the right metric to use for customer feedback.
The paper concludes that old-fashioned customer satisfaction and Net Promoter are statistically almost identical in their ability to predict customer retention, and Customer Effort performs somewhat worse.
Already, I’ve seen one NPS promoter claim that this “vindicates” NPS, which is not true. If anything, this research vindicates Customer Satisfaction, which NPS proponents often claim is less predictive than NPS.
But that aside, there are some important limitations to this research:
- The study was conducted using only Dutch participants. Given the fact that survey questions in different languages are literally different questions, the research isn’t applicable to English surveys.
- The overall sample size was respectable (over 8,000 ratings), but the follow-up survey to determine retention got about a 15% response rate. That means that a little over 1,000 responses were available to use for determining the statistical relationships between the metrics and retention. That’s sufficient (but not great), except that only about 20% of the respondents answered the Customer Effort question. So conclusions about the predictive value of Customer Effort are based on less than 300 responses total, a miniscule sample for this kind of study.
- The study authors also tried to see if there were differences between how the metrics performed in different industries, so they segmented the results into 18 (!) different industries. At the industry level, the sample is incredibly thin and the differences between the metrics generally slight, and I don’t see how the authors can justify trying to draw conclusions at this level.
Those critiques aside, this is an interesting paper and they did more right than wrong. I fear, though, that people who want to promote a particular survey metric are going to mis-use and mis-understand this research.
My own view is that way too much time and effort is spent arguing about what’s the “right” metric, when it’s far more important to have a robust process. If you get the process right, even a mediocre metric will give better results than a great metric and a terrible process.
So by all means read the research and understand the value of different survey metrics. But when you go to build your own program, spend your time making sure you follow the principles of Agile Customer Feedback rather than trying to find the perfect survey question.
