Just because your customers say it isn’t important doesn’t make it so

The intangibles matter. Don’t let bad research ruin your customer experience.

Bad research can wreak havoc with your customer experience. It can lead you to ignore a critical moment of truth while working on something with less impact.

In one journey map project, we were hired to extend a Big Research Company’s research. They had created an (ugly) journey map, and we were asked to replicate the findings in a local market. Which meant we had to use their methodology.

The way they conducted the research (and still conduct research today – this terrible method is rampant in journey mapping) was to ask customers to rate the importance and satisfaction of each touch point. The touch points with a significant difference between importance and satisfaction were “moments of truth.”

What a terrible idea. First, moments of truth are interactions with disproportionate impact on ongoing loyalty. You can’t discover them with this method. Most companies aren’t terrible at moments of truth – they just aren’t as good as they need to be. In addition, moments of truth aren’t always identified as important. It’s not whether they’re that important at that point – it’s whether they have long-term impact. Nearly every moment of truth we have discovered would not be identified using this method.

But just as important is the mistake of asking customers to rate the importance of steps in the journey.

This approach makes intuitive sense. After all, customers know what they value, right? Wrong. In fact, the effectiveness of simple stated importance has been debunked time and again. When you ask customers to rate importance on a 1-10 scale, two problems result.

First, it makes it appear that everything is important. On a 10-point scale, people will rate nearly everything an 8 or higher. That results in splitting hairs – the researcher is left to determine if the feature with an 8.5 importance is really more important than one with an 8.3.

Second, the one category of items typically rated low is intangibles – which are often actually more important to loyalty than tangible items. It is common to see items like follow-up phone calls, onboarding programs, and retailer welcomes receive low importance scores. But qualitative research (as well as quantitative research done correctly) consistently shows that these are some of the biggest drivers of loyalty.

So, how do you get a clearer view of your customer’s experience? There are two best practices:

1. Stated importance can work – but only if you introduce trade-offs. For example, Maximum Difference Scaling (MaxDiff) doesn’t ask customers to rate items on a 10-point scale. Instead, it shows 4 or 5 attributes at a time, asking participants to select the most and least important. This avoids the “everything is important” fallacy. But it still falls victim to the second issue – that intangibles are rated low. It also makes the survey much longer.

1. Derive the importance. A full description of this methodology is beyond the scope of this post, but Derived Importance Analysis is a statistical method that measures which questions most drive your important factors (satisfaction, NPS, delight, etc.). A simple way to look at this – if every promoter rates an attribute as a 5, and every detractor rates it a 3, then this attribute is probably important. This is overly simplistic, but helps to visualize how this works. Derived Importance has trade-offs, too. For example, if you’re doing consistently well on an attribute, so both promoters and detractors rate it a 5, it won’t come out as important.

While not perfect, both of these methods significantly outperform simple stated satisfaction.

What is shocking is that I see big research companies still using stated satisfaction, despite the fact that it often doesn’t work. My only guess as to why is that the intuitive nature of this approach leads them to avoid looking too closely at it, so they continue to use it.

Of course, there is a third option that should be used in conjunction with these others: avoid over-reliance on quantitative research. Any good quantitative approach must be combined with qualitative, and you need to look at where they differ. When developing New Coke, Coke received very negative feedback in their qualitative research, but very positive in their quantitative. We all know what happened as a result.

It’s not that quantitative research doesn’t matter. It definitely does. But you can’t rely on it exclusively. When the qualitative and the quantitative results vary, that’s when you need to pay attention. In our qualitative research, we consistently see the importance of intangibles. And we see the same result when using Derived Importance, which is one of the reasons we use this approach.

Your customers’ needs are complicated. Using a simplistic approach doesn’t work.

Bottom line: Never rely on one method to truly understand your customers’ needs. It can lead you to focus on just the tangible features, when intangibles are often the most important.