The Only Statistical Analyses You Need to Use On Customer Feedback Data

Statistical analysis is a big, complex and fascinating area of study. Okay, okay maybe it’s not fascinating for everyone and I can already hear a few yawns at the back of the room. But the good news is that if you are analyzing customer survey feedback there are just a few key tools that you need to master.

Get those sorted and you’ll be way ahead of your peers and really understand what your feedback is telling you.

So let’s step through some of the big statistical topics needed in customer feedback analysis.

What Sort of Data Do You Have?

Most of the data in customer feedback is called Ordinal data. Ordinal data can tell you the ranking of a value but the distance between the ranks is not clear. That is, you know that a 1 is ranked lower than a 2, which is lower than a 3 but the difference between 1 and 2 and 2 and 3 may be different. For example:

How responsive is Company X in closing the loop after problem resolution? Where 1 is very unresponsive and 5 is very responsive

The difference between 1 (very unresponsive) and 2 (unresponsive) is not necessarily the same as between 4 (responsive) and 5 (very responsive). That means you can’t say that a 4 (responsive) is twice as responsive as a 2 (unresponsive).

This is kind of a problem because, technically, this dramatically limits the statistical techniques that you can use with the data. For instance you, technically, can’t even calculate an average (or mean) value of the responses.

Now in real life we know that people average these sorts of scores all the time. But many of the statistical tests that we use on customer feedback data cannot, technically, be used on Ordinal data at all. Luckily for us though, while it is technically wrong it is generally accepted that there is still practical value in using these tools.

Additionally this well cited paper, Is the Selection of Statistical Methods Governed by Level of Measurement?, indicates that on practical basis, when testing for changes in mean, using this type of data is perfectly legitimate.

So if anyone says you “can’t use a t-test on Ordinal data”, you can tell them that you know it’s not technically correct but it is practically appropriate.

Do You Have a Big Enough Sample?

Sample size (how many data points you have collected) and confidence internals or standard errors are different sides of the same coin. Generally, yes there are lots of caveats, the larger the sample size the better you will be able to describe the population.

The concept of sample size comes up a lot surveying and it is an important idea but it’s significance (pun intended) is not nearly as large as it is commonly perceived to be in customer feedback.

This is because in almost all customer feedback work you are not looking at absolute values but relative values. The idea of an absolute measure of customer satisfaction, or Net Promoter Score, is nonsense. All you can really measure is relative customer satisfaction and movements in scores.

Sample size and confidence internals are very important when dealing with absolute quantities: how a group of people will vote, the variation in diameter of a wheel bearing. In these cases being able to closely characterize absolute values means something real. The absolute value of customer satisfaction does not.

Sample size does become useful in our customer feedback process when we want to test our hypotheses. The question “has customer satisfaction for this product increased?”, means something real. Here a larger sample size allows us to identify smaller changes in the relative scores and this is useful.

Questions That You Want to Answer

In customer feedback analysis there are only a few key questions you really want to answer and because the data types are pretty consistent the techniques you use are also pretty consistent.

Is the Score Different?

This question has a couple of versions:

1. Is the score different in this time period than last time period?
2. Is the score different for this group versus that group?

It doesn’t matter what score you’re talking about (responsiveness, cleanliness, Net Promoter, customer satisfaction, etc.) the question is always the same: Does this population give a different response to that population?

You want to know this so you can decide if an intervention you have devised, or lack of intervention has had an impact on the business, or perhaps an external factor has impacted on customer perceptions.

There a few of tools that you can use for these tests. The most common are:

1. Student’s t-test
2. Confidence Intervals
3. ANOVA
4. Chi-Square — this is used ot tell if the distribution of the score is different rather than the mean

Does Changing this Cause That to Change?

Here we first need a little stats terminology.

1. Dependent Variable: In customer feedback the dependent variable will normally be a proxy for an important business measure such as Net Promoter Score, Customer Satisfaction, “Would recommend” score. If you are lucky you will have access to the direct business measure for the respondent: share of wallet, revenue, profit, etc.
2. Independent Variable: Typically these are the attributes that you are asking about in your survey: responsiveness, cleanliness, professionalism, colour, etc. They might also be attributes of your survey population: big business, manufacturer, employee numbers, etc.

Here the big question is “does a change in this attribute, change that outcome variable?”. You need to know the answer to this because before you can change the outcome (dependant variable) you need to understand what drives it (which independent variable).

It turns out that this is a fairly common question that statistical techniques can help us answer. In fact, there is a whole suite of Generalised Linear Models available for unraveling the interactions of independent and dependant variables.

As you can see, from the vast expanse of statistical techniques we have slimmed them down to just a handful of approaches that are, relatively, easy to get a handle on.

Do you use techniques that I haven’t mentioned here? Let me know in the comments section below