Net Promoter and ACSI Smackdown

9
2,080 views

Share on LinkedIn

There’s a lot to like about the Net Promoter Score that Fred Reichheld developed. Business executives are jumping on it as an easy to implement metric of customer loyalty that ties to business growth.

But academic research is finding fault with the methodology.



Tim Keiningham of IPSOS Loyalty emailed me about a paper he co-authored in the journal of Marketing. You can download it here.

According to Tim, there are two key findings from their research:



  1. Net Promoter is not a good predictor of growth at all
  2. They found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, they were able to replicate a subset of Reichheld’s reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Their findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI.

I recently interviewed Claes Fornell, Director of U of Mich National Quality Research Center, and founder of the ACSI method, who said the NPS approach was “full of statistical errors.”



Is Reichheld’s claim that NPS is the “one number you need to grow” built on solid statistics, or not?

9 COMMENTS

  1. Thanks, Bob, for sharing the negative report on NPS.

    NPS is not perfect, of course. It is poor on diagnostics–it doesn’t guide us on exactly HOW to remedy bad NPS scores.

    But are some disagreements about statistics and correlatioins between growyh and profits (stats dreaned up by accountants) reason to throw NPS overboard?

    Would we throw out the blood pressure test or tenperature readings because they dom’t cure cancer?

    I believe NPS simple metrics have encouraged thousands of executives to measure the impact of customer experiences. And that is a positive step indeed–thanks to Fred Reichheld.

  2. Good points, Jay, but Reichheld’s claim is that NPS is the “one number you need to grow.”

    If NPS is not such a good predictor of business performance, as some academics now claim, then executives should not place so much faith in this metric.

    Bob Thompson, CustomerThink Corp.
    Blog: Unconventional Wisdom

  3. Jay —

    This is far more than a simple stat discussion. Our research found very strong evidence of research bias. We expect published research to be free of bias in management science, just as we do in all other fields of study. Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no HBR paper introducing Net Promoter without the research. Our research clearly calls the claims of Reichheld’s research into doubt.

    The core of your argument is that NPS got companies to focus on customers, so the linkage to growth isn’t that important. First, if it takes a flawed statistic to get companies to focus on customers, then they have bigger problems than customer loyalty. Second, if managers feel misled, the backlash will be far worse than any temporary gains, damaging the credibility of the importance of building customer loyalty to skeptical managers. Finally, Reichheld makes clear in his HBR article introducing Net Promoter that the “real test would be how well this approach explained relative growth rates for all competitors in an industry – and across a broad range of industry segments.” This statement makes it crystal clear how important this linkage is to the purported “reason” for Net Promoter.

    Regardless of our opinions with regard to Net Promoter, however, we must demand that the research published in prestigious management journals that is used to support it be unbiased.

    — Tim Keiningham

  4. Thanks to Tim for bringing the statistical issues to light. And without intending to “pile on,” I’d like to add another critical view of NPS. Attempting to predict any form of human behavior in such a simplistic manner reduces people (including customers) to, well, simpletons. The issue of whether or to what degree customers will recommend a product has more moving parts than a Swiss watch.

    So, while it’s nice to dream about shortcutting comprehensive customer research with a “down and dirty” alternative, it is still a dream.

    Dick Lee

  5. Increasingly, c-suite executives are pressuring marketing, marketing research, sales, and customer service to adopt metrics that are tied directly to bottom line results. Further, they want this proof in an easy-to-swallow tablet if at all possible. To Jay’s principal point, that’s the allure of having a single, simplistic ‘holy grail’ dependent variable.

    Those of us in the customer research community have been railing against NPS since Fred Reichheld’s article first appeared in HBR almost four years ago. There are scores of articles and white papers identifying its many deficiencies, mine (in this portal) and Tim’s among them. We feel, often, like we’re defending the Alamo against Santa Anna: our cause is right, but the opposing forces are strong and entrenched. Trying to discuss the cold realities of NPS with true believers, who prefer to swallow the pill rather than consider from whence it came, is like endeavoring to convince members of the Flat Earth Society that the Earth is round. The fact that, on multiple levels and with a portfolio of contra proofs, NPS’ problems render it inactionable and almost useless does not dissuade (or, at least, has not dissuaded) its supporters in the slightest.

    Only by giving results like Tim’s as much air and visibility as possible, and on a sustained basis, will those who are enthralled with NPS reconsider the wisdom of trying to address all of their customer loyalty behavior isssues with one number.

  6. I have followed the debate about NPS with interest, indeed I wrote a piece published on this site entitled “Why One Number is Not Enough” last year. There are two issues here, one which receives much air time, the other, little.

    Whether one number is sufficient to measure anything complex is of doubt. Would you trust an airline pilot who in her pre-flight announcement said that today they would only use the altimter when flying the plane? Would a CFO manage a company’s finances using only profit? Whether a unique correlation is there or not (and I doubt it), real insight never comes from one number. Jay is right in stating that NPS has helped promote the case for customer feedback on the executive agenda and that is to be applauded.

    The debate about statistics is of importance but the debate that receives insufficient air time is about how feedback is used. I have witnessed many executives rush to implement feedback systems with insufficient thought about how the data will be used. Its a bit like buying a car and leaving it the garage – you spend the money but don’t get the benefit. In fairness to Reichheld, he points this out in his book.

    If NPS scores improve, it is probably because the changes are being made to improve the quality of the experience. That is why feedback is gathered. That is how experiences are improved, which in turn leads to improved retention and advocacy opportunities that turn into revenue growth. It requires action at the level of individual customers (retention happens one customer at a time) as well as on the systemic issues that affect a wider group of customers.

    The feedback community should point out weaknesses in measurement systems but let’s also keep on banging the drum about turning feedback into improvements.

    Regards

    Dave Jackson
    http://www.clicktools.com

  7. I came upon this thread by searching Google for “problems with NPS”. The foregoing comments are thoughtful and thought provoking. Thank you.

    NPS was initially appealing to me because of the prospect of being more straightforward than traditional ‘customer satisfaction survey’ techniques. Older techniques seemed increasingly number exercises that lead to numbness among business leaders. It seemed less important that the clam of “one number” was hyperbole in a complex world of customer behavior as pointed out above. There are too many factors that influence growth to give any serious consideration to a single number claim. For instance, a business that sells products exclusively for new home construction probably grew very nicely between 2002 and 2005 … last year and this year growth would be much tougher regardless of what NPS score was being achieved.

    The more fundamental question is whether NPS is a good indicator of the company’s value dialog with the customer. Simple put: what’s the product or service worth [value creation], what value will the customer perceive and admit to, and what portion of that value will the customer yield to the vendor in the form of price and volume of business? Ultimately the value that is Captured and Converted to returns determines success – sustainable, profitable growth – not just how much value is Created.

    Regards,

    DJ Crane

  8. As the saying goes … “be careful what you ask for because you just might get it” … is so true when entering the wonderful world of NPS.

    What if there’s a problem with shipping?

    What if there’s a problem with product/service performance?

    What if the customer doesn’t like the format in which they receive invoices?

    What if the customer declines purchase because the price is to ‘high’?

    The answers to these questions are both procedural and political. And doesn’t NPS beg for a ‘system’.

    Hummmm ….

    Who are the internal stakeholders who will investigate and potentially resolve the feedback? Are they empowered to make change? … or are they merely a ‘super-delegate’ PR machine?

    How do these stakeholders manage these touches?

    How is the outcome data consolidated for a big picture scorecard?

    My company is putting the final touches on the development of a web-enabled, automated NPS system. Think of this technology in the world of lead management. NPS surveys conducted via on-line tactics have a direct umbellical cord to to this NPS technology. Those NPS surveys requiring resolution ‘follow-up’ are automatically routed to the pre-assigned NPS stakeholder who ‘owns’ the negative feedback issue. Within this technology, the stakeholder is given the background information of the NPS feedback … as well as the tools to manage follow-up activities. (Touch dates, call back dates, etc.) Additionally, there is a consistent NPS follow-up disposition form that will create dashboards of status and resolution successes/failures. We also examined any business demographics and sales history to profile the characteristics of NPS results for pre-emptive strikes in internal improvements. Additionally, we use the NPS data within target segments to tweak advertising and marketing messages.

  9. The NPS research is not based on any scientific rigor. Tim’s research on the NPS is excellent and pokes holes in the NPS claims that cannot be filled with any new research by the NPS camp. I applaud Tim’s research and his unapologetic stance regarding the problems with the NPS. Keep up the great work.

    Bob E. Hayes, Ph.D.
    Business Over Broadway

LEAVE A REPLY

Please enter your comment!
Please enter your name here