Here’s a fact: Survey question bias gives you skewed data. Another fact? Biased surveys are everywhere.
Survey biases can be obvious but often subtle, and survey writers can be oblivious to them entirely. So, how do you make sure your survey is bias-free?
Survey Question Bias: What It Is
As my friend and I were ending lunch at a chain restaurant, our server brought the check. At the bottom, the receipt included a QR code for a survey.
I sympathize with the server, but this won’t result in objective customer service data. When employees push for survey answers that benefit them or make their bosses look good, that’s a clear example of survey question bias.
Survey bias happens whenever questions push customers to report higher satisfaction than they truly feel.
Or the reverse can happen, too.
When customers are tired of a survey that feels like it will never end and the questions don’t make sense anyway, some customers will start assigning low scores simply to express their displeasure with your survey.
Bias undermines the point of a survey, which in the restaurant server example is to measure the quality of customer service.
Ideally, you want to know if your customers are unhappy. Inversely, if your customers see you as a ‘look no further’ company,’ i.e., they see you as having little room for improvement, you should want to know about that too!
Don’t Want to Read? Watch the 1-minute Video
Interaction Metrics’ Kaitlyn Bartley breaks down three of the most common survey question biases and explains how to fix them.
3 Types of Survey Question Bias
At Interaction Metrics, we’ve examined hundreds of clients’ surveys to make them more actionable and bias-free. Here are three types of survey question bias based on mistakes that we often see:
 #1. Double-Barreled Questions: These questions combine two separate issues into one. For example, don’t ask, “Was your rep efficient and proactive?” What if they WERE efficient but NOT proactive? |
 Remedy: Make sure each question asks about one aspect of the customer experience at a time. |
| #2. Forced or Irrelevant Questions: These questions ask customers to give answers about aspects of their experience they either didn’t encounter or don’t have an opinion about.
This happened to me with a Lowe’s Survey, which asked about ‘competitive product prices,’ ‘availability of desired delivery options,’ and ‘delivery tracking tools’ even though I had nothing to say about these matters. Here’s another example: Recently, I purchased running shoes in a store, but the post-interaction survey asked mainly about the online process. Even worse, the questions about the online experience were required; I had to put something in just to continue with the survey! Forced or irrelevant questions show customers you don’t care and aren’t listening. It’s annoying and skews your data! |
| Remedy: Use logic gating to keep your survey short and your questions germane. |
| #3. Leading Questions: One of the most prevalent types of survey question bias is the use of leading questions, where surveys prompt customers to give specific answers.
For example, “How satisfied were you?” assumes the customer was somewhat satisfied. Or “How likely are you to recommend?” assumes the customer is somewhat likely to recommend. |
| Remedy: Work with an outside (3rd-party) company to examine your survey for leading constructs. If you can’t afford this, at least ask other departments (who are not invested in your survey) to collaborate with you.Explain what leading questions are and why you are looking to avoid them. Perhaps their impartial eyes will help you to identify problematic words and phrases in your questions and answer options. |
Isn’t Healthcare Supposed to be Scientific?
Because healthcare organizations are supposed to be evidence-based and scientific, it feels particularly egregious when their surveys have biases. If you are in healthcare, I recommend this article because it explores bias in healthcare surveys. It points out that some common biases are question design, formatting problems, and how healthcare practitioners administer their surveys.
Additional Resource: Read our in-depth explainer to see the vast array of survey question biases pertaining to all kinds of surveys.
Bad Surveys, Good Surveys
Business strategies are only as good as the data they are based on.
If you’re using biased data to guide your business, you’re flying blind, making decisions based on guesses, not science.
You can write a bad survey in five minutes. But will it yield helpful information? Probably not. Writing a good survey that will give you reliable, actionable insights takes time, rigor, and careful design.
The Evolutionary Reason for Bias
The late psychologists Daniel Kahneman and Amos Tversky, who introduced the term ‘cognitive bias,’ wrote copiously about how countless biases are at the root of many errors in human judgment.
One of the most recognizable forms of bias is confirmation bias, where we seek out and recall information to bolster our preexisting beliefs.
Millions of years ago, confirmation bias may have been beneficial because it helped us make quick life-and-death decisions. Furthermore, confirmation bias may have enhanced group cooperation and cohesion.
Protocols
If biases are ingrained in our biology, this explains why they are difficult to spot in ourselves. The takeaway is we all need clear protocols and processes to detect and remove biases in everything we do.
Returning to customer surveys, consider each question carefully to stop the skew!
