List of the 20 MOST Common Survey Flaws

5
366 views

Share on LinkedIn

Picture Source: Interaction Metrics

This morning, my rep texted me from the shop where my car is serviced:
“You’ll receive a survey from us. It would mean a lot to me if you completed it. We hope you recommend us.” I get what he wants, but this is NOT how to do a survey.

Essentially, my rep is pushing for the answers he wants his higher ups to see. But that’s not scientific data collection. It’s public relations or marketing. Either way, it’s what we call Gaming and it’s the second flaw on the list below.

Since I frequently negotiate with my rep, I have nothing to gain by taking this survey truthfully.  Basically, by giving my rep a good score, I am more likely to win on future fees, and that’s my main concern. It doesn’t matter that I find my rep rude, I’ll give him a top score to stay on his good side and get the best pricing.



But is the point of the survey for customers to get good pricing? Or is it for the company to measure the quality of their customer service?

Why Issue Surveys? 

Why ask your customers to take surveys if the data you collect fails to capture the facts? Even worse, why issue surveys if the net result could be that your customers appear happier than they are.

Bad Customer Surveys are Ubiquitous

Earlier this year I wrote about how Whole Foods’ point of purchase receipt suffers from a complicated, disorganized survey request. I also explored how Alaska Airlines is pestering its customers with surveys that extend to 94 (!) questions.

Moreover, in 2016, we conducted a study of the surveys issued by the nations’ largest retailers. With one exception (7-Eleven), the surveys by the top retailers were unscientific and failed the basics of customer engagement. Learn more about the study and the criteria we used to evaluate the surveys here.

Basically, bad customer surveys are everywhere. In fact, they appear to be a multi-billion-dollar industry.

Imagine a World with Better Surveys

Let’s imagine a world in which companies ask meaningful questions in meaningful ways. Companies would only use surveys to interact authentically and learn from their customers. They would put a kibosh on leading questions and never bug customers with irrelevant or rote inquiries.

Toward this end, I have compiled some of the checks we use to vet the customer feedback programs that we manage for our clients. I divided this list of 20 flaws into 4 main types of survey errors. Let’s get started!

Bad Data from the Start

Have you ever had a contractor bring a washing machine to your house and show you a card that says, “Please give me 5 stars my job depends on it!”? Or maybe you bought a big-ticket item and were told they could come down in price for a good review? Garbage in. Garbage out. If you think your company may be collecting flawed data, develop a ‘how to survey’ strategy. Start by using mystery shoppers to document which customers are asked to take your survey — and how.

  1. Mis-Representative Samples
    This is when your survey data only comes from certain kinds of customers and doesn’t represent your customers at large. For example, your data could come from those with lots of free time, those with a specific gripe, or those who gave their email addresses. Basically, with mis-representative samples, your survey sample omits one or more customer groups—or your survey sample is too small to give you reliable facts you can count on.
  1. Gaming
    This is when associates only survey customers they believe had positive experiences. They also might ask customers to answer the survey in exchange for implied or explicit favors.


Poor Question Wording

How questions are phrased has everything to do with whether you will get accurate data and whether your customers will engage with your survey. The best way to clean up wording flaws is to work with an objective team, ideally outside your company. But if that’s not possible, at least work with another department.

  1. Leading Questions & Statements
    These questions prompt customers for the answers you want to hear. Both tone (when done over the phone) and content are used to push customers toward particular answers. For example, “How satisfied were you?” assumes the customer was somewhat satisfied.
  2. Double-Barreled Questions
    These questions ask two things at once, so it’s unclear which question the customer is answering. For example, “Was your rep efficient and proactive?” asks about two different behaviors in one question.
  3. Forced Questions
    These questions ask your customers to say something about themselves that may not be true. For example, “What’s your favorite gym?” when the customer may not use a gym.
  4. Unclear Wording
    This includes bad grammar and confusing language, anything that makes your questions unclear. In addition, words like “always” or “never” rarely apply to customers’ real circumstances and therefore should be avoided. Edit, proof, and test and then edit, proof, and test again.
  5. Excessive Use of Required Questions
    Too many required questions forces customers to answer questions that don’t apply to them, or that they simply don’t want to answer. This increases survey abandonment and nonsense answers.
  6. Under OR Overused Text Fields
    When your survey does not provide enough text fields for customers to explain their answers, valuable information is lost. Conversely, too many text questions is overwhelming and leaves customers mentally exhausted.
  7. Jargon
    Customers are usually not familiar with your company’s internal language. Using internal terms leads to skewed data as customers may interpret questions in ways you didn’t intend.
  8. Questions the Customer Can’t Answer
    These are questions that are best answered by someone besides your customer. For instance, we’ve seen the question “rate the white space to text balancing”. This is an example of a question best answered by a design team.
  9. Insufficient Use of Logic Gating
    Customers have a limited amount of energy they will spend on your behalf. The longer the survey, the lower the answer accuracy and completion rate. Good surveys use logic to maximize question relevance and minimize survey length.
    The Alaska Airlines survey mentioned earlier failed miserably with logic gating. In fact, Alaska Airlines punished frequent fliers by asking even more questions. Don’t overload your surveys. Instead, have a suite of surveys, each one used for a different purpose.
  1. Copycat Questions
    These are the questions like “how likely are you to recommend” that you see everywhere. Copycat questions might have a place in your survey but make sure that’s the case. Presumably, you want your questions to show you are truly listening and interested in gathering specific insights—right?!
  2. Limited Answer Options
    If the answer options don’t cover the full range of scenarios, customers may provide answers even if none of the options apply. If your options are not exhaustive, include “other” or text fields.

Scaling that’s Out of Sync

Scaling flaws happen when there is no clear mid-point to your answer options, or the anchors or selections are off kilter in some way. Scales matter. Give them their due!

  1. Vague Scales
    These are scales in which customers don’t perceive a clear difference between two options. For example, customers may not see “Exceeded Expectations” and “Above Expectations” as fundamentally different, yet they are measured 1 point (20%) apart on a 5-point scale.
  1. Unbalanced Scales
    These are scales in which the middle value is not the neutral response. It’s important to give your respondents a neutral option so they aren’t forced to give a skewed answer. An advantage of Net Promoter Score with its 0 to 10 scale, it that it has a natural midpoint of 5.
  1. Non-parallel Scales
    These are scales in which responses on the negative end of a scale are not the opposite of responses on the positive end. Be sure to use parallel anchors so that your scales make intuitive sense.

Analysis Weaknesses

The point of a customer survey is to find out what’s going on for your customers on a daily, weekly, or monthly basis and figure out what to change. When there is no team responsible for finding the story in the data, all that data you’ve collected goes to waste. The remedy? Prioritize analysis.

  1. Calling Hunches Facts
    Sometimes differences that seem important could just be due to random chance—a finding which should be disclosed, or simply thrown out.
  2. Ignoring Variance
    When the analysis only looks at averages, not the distribution of scores, you’re failing to get a nuanced picture. Instead, segment your data to determine what kinds of customers score low/high, and for what products, in what situations?
  3. Nonsense Graphics
    This is when data is presented in ways that confuse the issue. Using a pie chart when the answers don’t add up to a whole or comparing unlike phenomena in a single chart are examples of nonsense graphics.
  4. Wasting Customers’ Comments
    Text answers must be tagged, coded, and quantified, otherwise, key insights are lost. Reading verbatims doesn’t work because you won’t get quantification or a visualization of your customers’ comments. Software-based text analytics are a great start. However, for your verbatim analysis to be accurate, it’s essential to use researchers at least once per month to edit that output and test your customers’ verbatims for emerging words and themes.

How to Survey

It takes time and effort to learn how to survey well. But by eliminating these 20 flaws, you’ll be on your way to more effective customer surveys–the kind of surveys that enable you to steer your company tactically and strategically.



If you’re not sure where to start, reach out! We’ll examine your survey for free. If it’s good, we’ll tell you! If we find flaws, we’ll let you know what they are. And if you want to talk more generally about customer surveys and the customer experience, reach out!

It’s time for companies to get serious about how they collect customer feedback—and for all of us to demand better customer listening! Toward the best customer surveys!

5 COMMENTS

  1. Martha.
    Great article and a very useful precis on good and not so good survey practices. I wanted to add a couple and maybe underscore a few more, based on our experience at providing point of experience feedback.

    Agree:
    1) Keep the survey as short as you can, and adapt for the use occasion.
    We have found that in out of home surveys where mobile phones or tablets are the input, then the maximum survey time before incurring hight drop-offs is about 2 minutes. This means about 8 questions. Any more than this is detrimental to completion rates. It is much better to rotate questions over time than force-fit an over lengthy survey

    2) Inaction from the survey data
    An old study by Gartner suggested that while 95% of all US businesses collected feedback – only 10% ever used it. As part of the planning process, both the content of what is being asked AND the consequent actions / resources need to be planned in…..otherwise, why bother

    Additional Considerations
    1) Deliver the request for feedback as close to the transaction or meaningful experience point as possible. There is no point asking me if the receptionist smiled, two weeks after the event. I cannot even remember what I had for breakfast, so don’t expect some miracle recollection. In fact accuracy and response rates go way down the further from the actual event (if this is what you are measuring)

    2) Dont ask for personal identifiable info if you don’t need it. In our work we know this decreases response rates by over 20%. The advantage of not knowing your dogs maiden name is you will get more data.

    3) Enable the survey on their terms ie. a device they are familiar with, an input technology they prefer and even a language they are more comrtable with. Feedback on the users terms always delivers higher response rates. There is more on how this can be done here – https://opiniator.com/

    Again – great article – Thanks

    Matt

  2. Great insight Martha! Gaming is one that can be particularly “insidious” as survey responses might be what you want to hear, but are actually not reality.

  3. As a professor, I take surveys in part to note how bad they are constructed. To make a few quick points, the local NPR station didn’t include the option of listening to the radio when asking about how I accessed the station. Many surveys assume that I have cable television when I don’t and that I prefer my cell phone to my landline. My wife and I have not merged our finances even after being happily married for 36 years so that I can’t answer the question about who takes care of the family investments. Finally, I never give advice about financial matters so that the question about whether I would recommend such companies is always “no”.

  4. Thanks Robert. I agree with you. The vast majority of surveys are poorly constructed. The question that follows is ‘why?’

    Like anything cultural, the explanation is complex but here are a few answers.

    First, because surveys are easy to execute, the assumption is that good surveys are easy to do. Not so.
    Embedding and testing logic, writing good questions, running analysis… these are all skills. It’s akin to Photoshop, just because you have the program does not make you are a Jony Ives level designer.

    Next: since many surveys don’t provide strategic direction, some executives assume surveys CAN’T provide strategic insights, so they treat their surveys as a task to cross-off, not a program to treat as rigorously as their accounting. They get a software platform and call it ‘done’.

    Lastly, to keep this from getting too long, a lack of appreciation for science and expertise has become our cultural norm. That’s scary. Science and expertise are the values that will save our species which means appreciation for these values ideally would be fostered though our educational system and then reinforced through company conduct and corporate decision making.

    To not sound dreary, but rather to end on a positive note, some execs DO believe in running good surveys and they make for great clients and keep Interaction Metrics (my company) alive!

LEAVE A REPLY

Please enter your comment!
Please enter your name here