Top

Just because your customers say it isn’t important doesn’t make it so

Jim Tincher | Sep 15, 2016 95 views 3 Comments

Share on LinkedIn

The intangibles matter. Don’t let bad research ruin your customer experience.

Bad research can wreak havoc with your customer experience. It can lead you to ignore a critical moment of truth while working on something with less impact.

In one journey map project, we were hired to extend a Big Research Company’s research. They had created an (ugly) journey map, and we were asked to replicate the findings in a local market. Which meant we had to use their methodology.

The way they conducted the research (and still conduct research today – this terrible method is rampant in journey mapping) was to ask customers to rate the importance and satisfaction of each touch point. The touch points with a significant difference between importance and satisfaction were “moments of truth.”

What a terrible idea. First, moments of truth are interactions with disproportionate impact on ongoing loyalty. You can’t discover them with this method. Most companies aren’t terrible at moments of truth – they just aren’t as good as they need to be. In addition, moments of truth aren’t always identified as important. It’s not whether they’re that important at that point – it’s whether they have long-term impact. Nearly every moment of truth we have discovered would not be identified using this method.

But just as important is the mistake of asking customers to rate the importance of steps in the journey.

This approach makes intuitive sense. After all, customers know what they value, right? Wrong. In fact, the effectiveness of simple stated importance has been debunked time and again. When you ask customers to rate importance on a 1-10 scale, two problems result.

First, it makes it appear that everything is important. On a 10-point scale, people will rate nearly everything an 8 or higher. That results in splitting hairs – the researcher is left to determine if the feature with an 8.5 importance is really more important than one with an 8.3.

Second, the one category of items typically rated low is intangibles – which are often actually more important to loyalty than tangible items. It is common to see items like follow-up phone calls, onboarding programs, and retailer welcomes receive low importance scores. But qualitative research (as well as quantitative research done correctly) consistently shows that these are some of the biggest drivers of loyalty.

So, how do you get a clearer view of your customer’s experience? There are two best practices:

  1. Stated importance can work – but only if you introduce trade-offs. For example, Maximum Difference Scaling (MaxDiff) doesn’t ask customers to rate items on a 10-point scale. Instead, it shows 4 or 5 attributes at a time, asking participants to select the most and least important. This avoids the “everything is important” fallacy. But it still falls victim to the second issue – that intangibles are rated low. It also makes the survey much longer.
  1. Derive the importance. A full description of this methodology is beyond the scope of this post, but Derived Importance Analysis is a statistical method that measures which questions most drive your important factors (satisfaction, NPS, delight, etc.). A simple way to look at this – if every promoter rates an attribute as a 5, and every detractor rates it a 3, then this attribute is probably important. This is overly simplistic, but helps to visualize how this works. Derived Importance has trade-offs, too. For example, if you’re doing consistently well on an attribute, so both promoters and detractors rate it a 5, it won’t come out as important.

While not perfect, both of these methods significantly outperform simple stated satisfaction.

What is shocking is that I see big research companies still using stated satisfaction, despite the fact that it often doesn’t work. My only guess as to why is that the intuitive nature of this approach leads them to avoid looking too closely at it, so they continue to use it.

Of course, there is a third option that should be used in conjunction with these others: avoid over-reliance on quantitative research. Any good quantitative approach must be combined with qualitative, and you need to look at where they differ. When developing New Coke, Coke received very negative feedback in their qualitative research, but very positive in their quantitative. We all know what happened as a result.

It’s not that quantitative research doesn’t matter. It definitely does. But you can’t rely on it exclusively. When the qualitative and the quantitative results vary, that’s when you need to pay attention. In our qualitative research, we consistently see the importance of intangibles. And we see the same result when using Derived Importance, which is one of the reasons we use this approach.

Your customers’ needs are complicated. Using a simplistic approach doesn’t work.

Bottom line: Never rely on one method to truly understand your customers’ needs. It can lead you to focus on just the tangible features, when intangibles are often the most important.

Print Friendly

Republished with author's permission from original post.


You Also Might Like...


Categories: BlogThink TankVoice of Customer

95 views

3 Responses to Just because your customers say it isn’t important doesn’t make it so

  1. Michael Lowenstein September 16, 2016 at 5:38 am (1142 comments) #

    There are two key research issues at play here. First is the ‘say vs. mean vs. do’ dichotomy. Whenever importance is evaluated on a direct basis, there is a tendency for the ratings to condense, often at the top, where lots of things are important. It’s critical to understand what customers really mean from an emotion and memory perspective, and then how they behave, i.e. what they do, in the marketplace. Next, as an expansion to the first point, rating direct importance has been regarded as non-actionable for years, and modeled, or derived, importance is little more than correlation analysis, that is simple regression, often around semi-actionable dependent variables like satisfaction, NPS, or CES.. This approach doesn’t address real contributing factors, i.e. causation. MaxDiff and structured equation modeling gets analysts out of the traps and limitations of these more antecedent methods.

  2. Shaun Belding September 16, 2016 at 5:43 am (40 comments) #

    Right on the mark, Jim.

    Customers don’t know what they like when it comes each step of their journey. Most people aren’t that pedantically analytical about their lives. Over the years, I have seen quite a bit of bad research that insists on drawing flawed conclusions from responses to meaningless questions.

    Quantitative research is best used to establish the personas and the actual journeys for each persona. To establish touchpoint importance and the true performance gaps within each journey, you need a lot of qualitative input.

    It also helps, when possible, to do an objective audit of known best practices. (e.g.: using mystery shopping to assess efficiency or friendliness of service).

    It’s a relatively new discipline, and I’m afraid that there are many journey mapping initiatives out there that are providing companies with little, no, and sometimes negative value.

  3. Valerie Peck September 16, 2016 at 7:55 am (2 comments) #

    Good point Jim. We like to do a 360 view of everything when we map. Just asking a prospect or a customer how it went only gets you a simple, and as you note, sometimes misleading viewpoint.

    The key is collecting DATA – really understand what actually happened. Data can be transactional, behavioral, social, survey driven (when scored).

    We find dissatisfaction is a lot more important than satisfaction BTW and clues like unsubscribes, attrition, drop off in the pipeline, verbatim all are great.

    Social sentiment – though broad and generally less focused gives you ‘color’ and what they really did – drop off, convert, etc. all is in the Data.

    Unfortunately most companies dont collect data at touch level these days.
    If you have Data you can do a linear regression to see what the actual MOT or Pain Point is as well.

    When we do Voice of the Customer we always correlate that with Voice of the Company and a Maturity Model to see what Operations/People/Data/Technology is working (or not) at each touch as well.

    We also think that ‘loyalty’ is emotional and inspirational – nice to have – what we go for is repurchase and tenure – thats where the payoff lies.

Add Your Comment (All comments are reviewed by moderator, no spam permitted!)