Top

Are You Asking Customers Too Many Questions? Try Just Two 

Bob Hayes, PhD | Sep 8, 2017 1,545 views 4 Comments

Share on LinkedIn

Customer feedback plays a central role in customer-focused programs. Last month, I argued that we need fewer questions in our customer survey. In this post, I will present a new customer survey method that only has only two (2) open-ended questions. I will use a case study to show that that these two questions provide comparable customer insights to longer, more traditional customer surveys.

The Goal of Customer Surveys

Annual customer surveys are the most popular ways of collecting customer feedback. These relationship surveys are typically used by business leaders to help make better strategic decisions. To help make these decisions, these leaders need three key pieces of information that these surveys provide:

  1. Current level of customer satisfaction – Surveys provide an index or metric that represents the level of satisfaction of your customer base. Before you start your company’s journey of self-improvements, you need to know how satisfied your customers are today. This metric can be used in any analytics efforts to study customer satisfaction.
  2. Context of customer satisfaction – Surveys provide context for the responses. It’s important to understand your brand in the context of your customers’ experiences. How customers describe the company or their problems helps highlight the underlying conditions that drive their satisfaction and dissatisfaction and could generate improvement ideas.
  3. Key drivers of customer satisfaction – Surveys identify the key drivers of satisfaction. When you invest in your company, you want to optimize the ROI by first improving business processes that reflect key drivers. Target improvements in areas that matter most to customers.

Comparing Two Survey Methods

I will examine two survey methods to compare the outcomes on the three criteria above. The first survey method is the traditional survey that contains 20+ questions, including loyalty questions, several CX questions and open-ended questions. The second survey method contains only two open-ended questions.

1. Traditional Survey (13 questions): The survey contains 3 loyalty questions: Overall satisfaction, Likelihood to recommend and Likelihood to buy again (0 to 10 rating scale). Also, the survey contains 10 CX questions that cover various customer touch points (e.g., product quality, technical support, ease of doing business) (0 to 10 rating scale).

2. New Survey (2 questions): The survey contains the following two open-ended questions.

  1. What one word best describes Company Name?
  2. If you were in charge of Company Name, what improvements, if any, would you make?

In a recent client engagement, I conducted a survey for a B2B technology company using all of the questions above. I analyzed the respective questions for each survey method to compare the results across the two methods. Because the two methods are based on the same respondents, I expect the results of the two methods will be similar. My hope is that the two methods would result in outcomes that are comparable.

Results

Below are the results of the two survey methods.

Traditional Survey


Figure 1. Customer Satisfaction metric based on ratings.

1. Customer Metric. We can calculate a numerical index easily simply by using the ratings provided by the respondents. As you can see in Figure 1, we can use this satisfaction metric to track changes over time. The results indicate that the level of satisfaction has remained relatively stable over three-year time span reported here.

If you are more comfortable using percentages in your dashboards, you can easily calculate percentages as your metric of choice.

2. Context. We can understand the context of the satisfaction ratings by examining the different touch points measured in the survey. Figure 2 contains information about the different touch points. Customers are more satisfied with product, technical support and training than they are with the web site, purchasing/invoicing/delivery and the sales process.


Figure 2. Satisfaction ratings of customer touch points represent the context of customer satisfaction.

3. Key Drivers. We can identify the key drivers by conducting what is traditionally referred to as Driver Analysis. The results of this analysis appears in Figure 3. As you can see in figure 3, for each of the 10 touch points, I have plotted both the satisfaction rating as well as its correlation with our metric of customer loyalty (aggregate of overall sat, recommend and buy again).


Figure 3. Driver Analysis showing the key drivers of loyalty.

The results of the driver analysis show that the key drivers were represented by two customer touch points: purchasing/invoicing/delivery and the sales process. These two areas represent the touch points that are both important to customers (as is indicated by the high correlation with loyalty) and in which the customers are not as satisfied as they are with the other touch points.

New Survey


Figure 4. Customer satisfaction metric based on the responses to the question, “What one word best describes this company?”

1. Customer Metric. The one-word question is used to calculate a sentiment (satisfaction) score. Using a sentiment lexicon (see here for the development of this lexicon), each word was transformed into a numeric value ranging from 0 (negative sentiment) to 10 (positive sentiment). This metric can be used to track customer satisfaction over time (see Figure 4) as well as test other hypotheses you might have (e.g., test differences across customer segments).

This method of calculating customer satisfaction (derived from word) appears comparable to the traditional method using ratings. Satisfaction remained very stable over the same time period. In fact, the correlation between the two customer satisfaction ratings for this survey was r = .52, showing there is some agreement between the two methods of measuring customer satisfaction.


Figure 5. Understanding context of satisfaction – Word cloud based on responses to question, “What one word best describes this company?”

2. Context. We can understand the context of the satisfaction metrics by simply examining the words that the customers used to describe the company. You can graph the frequencies of the words in a Pareto chart to highlight the different rates at which words are used. I also like to present the results in a word cloud to highlight the popular words that are used to describe the brand (see Figure 5).

This summary provides a high-level picture of how customers describe your company using their own words. Note that many of the popular words found using this question were not found by using the traditional approach of asking the customers to rate standardized CX questions. This approach revealed potentially useful words like, “Reliable,” “Innovative,” “Expensive” and “Leader.” Not only do these words provide you context behind what is driving customer satisfaction, they can also be used to improve how to better communicate your company’s value in sales and marketing collateral.


Figure 6. Understanding context of satisfaction – What improvements would you make?

We can also understand context of satisfaction by examining the responses to the second question. For this question, the respondents indicated what they would improve if they were in charge of the company. A content analysis of the responses showed that improvement areas (touch points) reflected a wide range of topics (see Figure 6). The most frequently mentioned touch points needing improvement were product, price and purchasing/invoicing/delivery. The remaining touch points were mentioned far less (all in single digits).


Figure 7. Key Drivers – Satisfaction drops in the presence of specific touch points.

3. Key Drivers.

To identify the CX touch points that were the key drivers, I combined the two pieces of information for each CX touch point into a single chart (see Figure 8). This is somewhat similar to the earlier driver matrix with the exception that, instead of looking at the impact of each CX touch point, we are focused on the magnitude of problem for each CX touch point.


Figure 8. “Driver Matrix” for two-question survey to help prioritize CX improvement opportunities

Based on this analysis, we see that:

  • Two CX touch points are considered key drivers of satisfaction (colored red): purchasing/ordering/delivery and customer service. That is, customers mention these touch points as needing improvement more than other touch points and they are less satisfied when they mention these touch points.
  • Dark green touch points indicate your customers are not mentioning them much and, when they do, they are still satisfied; these are not high priority to improve.
  • Light green touch points indicate many customer are mentioning them but, when they do, they are still satisfied. Improvements in these areas won’t have much of an impact on satisfaction.
  • Yellow touch points indicate few customers mention these areas but, when they do, they are less satisfied. Watch these areas carefully to ensure you are not underestimating the magnitude of this problem. For example, it’s conceivable that only a small portion of customers recently went through the sales process, artificially lowering the percentage of customers who said sales needs improvement. To be on the safe side, you might consider the yellow touch points as “next areas to improve” after addressing the red touch points.

Summary

I wanted to compare two survey approaches to determine if a short survey (two questions) would provide comparable results to a long survey (13 questions). Using the three criteria for good customer surveys, I found that:

  1. Customer satisfaction metrics behaved the same across both methods. The two survey approaches resulted in similar findings in that satisfaction was fairly stable over a few years.
  2. The context of satisfaction could be determined by both methods. The one-word question provided additional information about the reasons behind the satisfaction which was not gleaned from the longer survey.
  3. Key drivers (areas that would result in greater ROI) could be identified using both methods. The key drivers overlapped considerably across both survey methods (traditional survey: sales, purchasing/invoicing/delivery; new survey: customer service, sales, communications and purchasing/invoicing/delivery.

By using a short customer survey with the right questions, you can improve your customer experience improvement program. Here are five primary benefits to using the method described here:

  1. Save money. While the analysis of text responses does require extra work, that process is typically automated. Shorter surveys are more cost-effective to manage than longer surveys, requiring less resources on the part of your CX team and other interested parties.
  2. Obtain reliable, valid and useful information about your customers. Insights gleaned from the shorter survey are comparable to longer surveys.
  3. Hear from more of your customers. Hear from more of your customers. Customers are more willing to complete shorter surveys than longer surveys.
  4. Minimize customer effort. Short surveys provides customers a quick and easy way to tell you what you need to do to win their business. And short surveys show you respect your customers’ time.
  5. Identify operational best practices, track satisfaction and understand customer segments. The customer satisfaction metric can be used to track changes over time as well as test hypotheses about your customers.

Using two specific questions, I found that you are able to glean much useful information about your customers’ attitudes that can help improve your customer programs. Coupled with all the other information businesses have about their customers (think Salesforce, Google Analytics, Mixpanel, Marketo), a short survey can provide a comprehensive picture about the health of your customer relationship and help you determine an optimal improvement approach.

Are you asking too many questions of your customers? It may be time to simplify your survey.

Print Friendly, PDF & Email


Recent Editor's Picks:


Categories: ArticleColumnCustomer AnalyticsEditor's PickPerformance MetricsVoice of Customer
Tags:
1,545 views

4 Responses to Are You Asking Customers Too Many Questions? Try Just Two

  1. Western September 20, 2017 at 7:34 am (1 comment) #

    Great article Bob. While I was doing my MBA in Marketing I took a job as a marketing assistant at one of the cruise company. They used to have two to three page marketing surveys that were sent to their mostly 60 plus customers. The response rate was less than 2%. I asked innocently why we needed to ask so many questions and explained that if we knew what we needed to accomplish than we could ask five to ten questions as our audience are elderly so we should not burden them. Suffice to say I got canned the same day but fortunately my message got through and they started creating shorter surveys and the first survey they did got 20% response.

  2. Bob Hayes September 20, 2017 at 12:24 pm (22 comments) #

    Thanks, Western. That’s an unfortunate experience you described about your prior role. I wonder if their insights were similar to their longer survey. I suspect they are (given the intent of the survey was the same).

    We really need to reconsider the questions we ask in our surveys. If the analysis of the survey data tell you that you’re asking too many questions (where fewer questions would suffice), you need to stop burdening your customers, old or young, with unnecessary questions. Besides, as companies get deeper into the digitized way of doing business, they’ll have access to much more data about their customers that can be used to augment survey data.

  3. Dave Fish September 22, 2017 at 5:02 am (4 comments) #

    Bob

    Great study on how we could and should move to open ends and sentiment. There is one practical issue that does come out moving from scales to open ends for KPIs; end users may not trust them especially if they are used for incentivization. Will need to find a simple way to describe and show the reliability of such approaches in order to find wide field acceptance. That being said, this is definitely the way to go IMO. Thanks for writing such a succinct case study on how to make it happen.

  4. Steven Walden October 9, 2017 at 1:01 am (9 comments) #

    More stories like this fewer stories like that is my favoured KPI here. See cognitive edge and vector approaches. I am less a fan of regression analysis since not everything is past predictive

Add Your Comment (All comments are reviewed by moderator, no spam permitted!)