This post contains (almost) everything you need to know about customer survey design. When designing a customer survey, as with anything, planning is key. As the saying goes, if you don’t plan, you plan to fail.
It can be useful to think of the end-point and going backward when planning. What insights do we want to get? And how to design our survey questions so that we can achieve those outcomes?
To best design a successful survey, I want to introduce you to the 4 steps for successful survey design. A while back, I hosted a webinar with Dr. Jenine Beekhuyzen, author and Director of Adroit Research, on this exact subject, which I now want to share with you.
The 4 steps to customer survey design are:
- Plan: Plan for survey design success
- Survey Design: Receive authentic responses
- Collect: Get the most out of your survey
- Analyze: Find answers and insights in responses
First thing’s first, let’s start with planning.
1. Plan: Plan for customer survey design success
There’s no better time than now to start thinking about how happy your customers, clients, or your staff (or whomever you’re surveying) really are.
What we usually want to pinpoint are their positive versus negative experiences.
What is it that they actually want rather than what you want? We need to find somewhere in the middle.
Often, we go with certain assumptions about what we want to learn, but we also need to think about whether it’s all we want to know.
Or, do we want to give our recipients the opportunity to share their ideas? We need to get to know them as much as possible. Some people are shy, others are forward with sharing their ideas, so we need to create an environment where everyone feels comfortable.
Things you can investigate:
- How happy are your customers/clients/staff?
- What do they want?
- Get to know them and their needs.
- What do you want to know?
- Will your questions confirm your assumptions or explore new ideas?
Top tips for survey design planning
- Keep it short!
If there’s one top tip when designing your survey, it’s this: remember to keep your surveys short. Only include questions that you are actually going to use.
You might think there are lots of questions that seem useful, but they can actually negatively affect your survey results. Another reason is that often we ask redundant questions that don’t contribute to the main problem we want to solve. The survey can be as short as three questions – and that’s fine.
- Use open-ended questions first
To avoid enforcing your own assumptions, use open-ended questions first. Often, we start with a few check boxes or lists, which can be intimidating for survey respondents. An open-ended question feels more inviting and warm – it makes people feel like you want to hear what they want to say and actually start a conversation.
- Ask questions to reveal main problems
Pretty straightforward, your surveys will reveal what areas in your business need extra support or create bottlenecks in your service.
- Use surveys as a way to present solutions
Think about using surveys as a way of presenting solutions to your audience and getting direct feedback on those solutions in a more consultative way.
- When to send your surveys?
It’s important to think about the timing of your survey. Are you sending your survey at 8am in the morning when staff arrives, just about to start the working week?
Or are you sending it Friday afternoon, when people find they have less to do and perhaps are more inclined to respond? Consider when your audience is most likely to respond to your survey and give them the opportunity to do it at their leisure, at the time that suits them.
- Incentives drive 5-20% increase in responses
Oh yes, the big question – to incentivize or not to incentivize? There’s a lot of discussion around this. If incentives are done well, they shouldn’t skew the accuracy of the survey results.
Can prizes skew satisfaction scores? As long as your survey is consistent and the focus is not on the score itself, but on the open-ended question, including prizes should work.
It comes back to what’s in it for them (your respondents). What incentives would motivate your participants? Is this monetary, or is it having their say and being part of the conversation? Sometimes, that may suffice.
Incentives: How Airbnb does it
Here’s a great example of how Airbnb incentivizes its customers to leave feedback.
They basically say you won’t find out what your host thought of you as a guest until you write your review of the host. Clever!
If that’s not a strong incentive, I don’t know what is.
Airbnb encourages the customer to fill out the survey by showing a picture of the accommodation where they stayed, creating an emotional connection with the experience. Again, the customer is more likely to fill out the review.
Also, notice how they use the word “you” a lot. And, the way they’re writing in a very simple and concise way. Once you click through, they ask you to tell the next guest what you loved and anything else they should know about this place.
And then thanking them, super simple.
They don’t just use a lot of “you’s” but also “we” implying there is a relationship and a mutual benefit in reviewing. Closed and open-ended questions are mixed in a page without the need to scroll between them.
Using a star rating system is a very familiar tool for most people. Nowadays, you could try using to emojis instead of stars, which are even more familiar to younger people and millennials.
Question the questions
Interrogate the questions to identify superfluous questions which you could find the answer to elsewhere or that you aren’t actually planning to use. The most common culprits are people’s gender, age, background, Ethnicity, email address, how long they’ve been a customer and so forth.
Shorten the survey where possible!
And really think about the wording of your questions: Does it appeal to your audience?
This can make a huge difference to how people respond. Therefore, it’s recommended to pilot your questions with a small group of people, if you can.
You don’t want the scenario where people don’t respond simply because they didn’t understand a question or didn’t understand what is expected of them.
For good survey design – always ask:
- What insight am I hoping to get from this question?
- Is it likely to provide useful answers?
2. Survey Design: Receive authentic responses
Now, let’s talk about how to receive authentic survey responses.
There are several different ways to present the answers or the possible answers, as you can see in the image below.
If you have 5 response options, the intensity should increase from one end to the other. In the example on the left-hand side (see image), technically speaking “Slightly important” and “Somewhat important” are both subcategories of “Not very important” and the placement of these options can confuse respondents and lead to incorrect results.
You need to explicitly label each option as clearly as possible. So, in the example to the right we can see that “Very important” conveys less intensity than “Extremely important”, and more intensity than “Slightly important”.
Example of two survey designs
- Open-ended questions = more insightful answers
- Closed questions = easier to respond to, easier to analyse
- The best approach – use a mix of questions
- It’s more compelling to answer different types of questions
- Multiple text boxes can be intimidating
I was recently invited to fill out a survey where I was presented with a wall of radio buttons. Now, as a respondent, it made me worried that I’d select the wrong row and I didn’t feel the buttons made a lot of sense (see image below).
The additional column at the end, which does not “increase the intensity” of the response but instead presents two alternative options, were confusing.
Using scales questions correctly
Key risk: Confusion and inaccurate data
When it comes to scales, consistency is absolutely key. When people are faced with the example above, where you have a so-called Likert scale (a scale used to represent people’s attitudes to a topic), but then you have another option or option, it can be extremely confusing, as can mixing up different scales.
Use the same scale throughout
So, think about which scale suits your organization? Then stick to it, so that when we survey people, again and again, they are used to our formula already. In the Likert scale 1 to 5, 5 should always be “good” and 1 should always be “bad”, again for consistency.
Explain the meaning (is 5 good or bad?)
When it comes to scales, here are a couple of other examples. Dichotomous scales are very black and white, there is no room for “maybe” or “neutral”. In rating scales there is a range, so on a scale from 0 to 10 we’re trying to find out how people are feeling, and it allows for a range of answers.
Semantic differential scales, are useful to identify people’s attitudes. They can be a bit harder to analyze, but in terms of data capture, they give you the most nuance and the most breadth.
3 Types of Scales Questions
- Yes/No, Agree/Disagree
- Allows for no nuance
- Rating Scales
- CSAT and NPS “On a scale from 0 to 10, …”
- Likert scale: 1 (bad) to 5 (good)
- Allows for a range of answers
- Semantic Differential Scales
- Best for attitude evaluation
- The most nuance
Challenge your assumptions
It’s crucial to challenge your assumptions, as it’s very tempting to make assumptions about why things are the way they are. For instance, why some teachers are excited about certain educational technology in schools, and others aren’t. There is usually more than meets the eye about a person’s background which can affect the scenario.
Jenine says: “We try to collect data from as many people as we can to get different perspectives. Of course, that’s not always possible, but at least it helps to understand why we’re getting the responses that we are”.
Have multiple survey-writers
To have multiple survey writer can be helpful here, as having people read each other’s work and test the questions helps address the fact that most questions can be interpreted in more than one way.
Set reasonable expectations
Jenine explains: “We should try to encourage the reluctant people to participate as they have a very interesting perspective that should be included in the overall data capture. We want to get a reasonable response rate so that we can feel comfortable and confident in the results”.
Choose your survey questions carefully
When you’re choosing your survey questions, make it really count. Only use those that can make a difference to your end outcomes.
Be prepared to report back results and take action
There are so many times I fill out a survey and never hear about it. As an incentive, you can share the results with the participants, in the form of a benchmark, or a measurement that you then report to the participants.
3. Collect: Get the most out of your customer survey
How do we make sure that we get the most data and get the most out of this data?
Collect data from a wide audience
Think about your entire sample: who is part of it and who are the stakeholders. You might need a different plan to engage different stakeholders depending on their role.
Again, it comes back to “what’s in it for me”. If as a respondent I don’t see the value, I’m not going to participate. Whether it’s reporting back or whether it’s seeing what other people have contributed.
Collect data that can’t be directly observed
We’re collecting data that we can’t directly observe. You could have informal conversations with people, but it’s difficult to measure these conversations, which is why we collect survey responses.
Often what people say and what people do differ, which is why mixed methods have become a popular approach to research. Respondents are going to tell me what will happen, but what actually will happen may be different.
Always have more than one data source
Make sure to use more than one data source. So, you have the survey, you might also have a chat log and some other way of collecting qualitative data.
Survey, and then survey again
It’s a good idea to survey different people at different points, for example, dissimilar customer lifecycles and particular times of year and so on.
Good questions to ask could be:
- What’s the point of our research right now?
- What will it look like in the future?
- How do we present it in a cohesive manner to the entire organization?
When to survey and when to interview?
This leads me to the big debate: Should you interview or survey people? As mentioned, for some people, we need a different approach. At Thematic, we’ve noticed that some companies prefer to use interviews rather than surveys. The interviewer either types in the responses into the survey software, or they record and transcribe the interviews.
This is a matter of breadth vs. depth. Interviews allow you to contact a smaller number of people but in a greater detail, survey data allows you to explore a wider audience.
Jenine says: “I would recommend based on the published research and the many projects I’ve led, to start with an open-ended question in a survey. Once we’ve asked them what their concerns are in an open way, we can then design a survey to measure these concerns”.
So, if you measure “professionalism”, you could interview people to learn what specifically they include under “professionalism” in this context. As an idea, you can pose these as tick boxes in another survey to measure their importance.
How can an interview best augment a customer survey?
The question being: How can interviews augment a survey as part of a wider research process? Remember, that an interview can be just 2 or 3 questions – short and concise. Often, we ask very similar questions in the survey, but we get different responses because of the format. It helps to use the same questions, because different ways of getting the insights appeal to different learning styles of different people. People who are comfortable sharing their ideas in a survey, may be uncomfortable in an interview.
Who should interview?
The person who put together the questions and knows the rationale behind them, is the person who should conduct your interview,
Who should they be interviewing? I would suggest finding key informants within the sample who would give us a range of views. You need to play devil’s advocate and get responses across the spectrum, so that you can get a range of survey ideas.
How to make it compelling to drive the most insights?
It can be challenging to make it compelling, but key points are to use simple language and use examples. Images work well here. To make it super easy, you can provide samples or examples of responses of the types of answers you received in previous surveys.
How to get deeper insights?
To ensure consistency, we want them to answer the way they want to, but not force them to respond with certain things. It takes a certain skill to design such an interview. Practice, practice, practice!
What survey collection software should you use?
Here’s a list of software options that I’ve came across.
- Google Forms: is free and straightforward to use. Collects the results into a Google spreadsheet.
- SurveyMonkey, SurveyGizmo and TypeForm: inexpensive, user-friendly and they allow you to personalize the survey invite. You can upload a spreadsheet with customer data, from which you can use variables when writing an email, to make it seem more personal.
- Zendesk and Intercom: tools for communicating with customers and have options for surveying customers.
- Qualtrics: a very popular survey platform that’s been around for many years. It has specific tools for certain types of surveys, such as Promoter.io, AskNicely and Delighted, which are used for NPS surveys. The benefit of these surveys is that they provide a dashboard through which you can see each customer’s response and can follow up with them. The downside is that if you want to change your survey to include additional questions, you’re stuck because these tools aren’t easily configurable.
Also good to know, there are open-source tools for designing your own surveys.
As an example, I recently came across this popup survey while working in our internal company wiki by Atlassian. I liked that it wasn’t intrusive – once I clicked on a number, an open-ended question field appeared. Again, non-distractive, and the pop-up appeared in the same context, I didn’t need to leave the page. There are open-source packages that allow you to implement similar surveys if you have access to development resources.
4. Analyze: Find answers and insights in responses
Now, you need to analyse the collected data to find answers to questions and insights. This is what we do at Thematc: we analyze customer feedback to tell companies how they can increase customer satisfaction, increase loyalty and decrease churn.
Survey design questions to interrogate the data:
When we work with customers, we always recommend starting with questions. Here are some example questions that help interrogate the data.
- What are the most common responses to questions X?
- Which responses are affecting/impacting us the most?
- What’s different about this month/this year?
- What did respondents in group Y say?
- Which group of respondents are most affected by issue Z?
- Have customers noticed our efforts in solving issue Z?
- What do people say about Z?
For example, look at question 1 and 2. The difference between the two is that the first one returns the volume, whereas in the second one we can look at the volume relating to a particular satisfaction score. If something is very common, it may not affect the score. But if, for example, your Detractors in an NPS survey mention something a lot, that particular theme will be affecting the score in a negative way. These two questions are important to take hand in hand.
You can also compare different slices of the data, such as two different time periods, or two groups of respondents. Or, look at a particular issue or a theme, and ask questions such as “have customers noticed our efforts in solving a particular issue?”, if you’re conducting a continuous survey over multiple months or years.
But of course, getting insights from responses is difficult, especially if you use a lot of open-ended questions. Why is that? Because what comes back is a wall of text, which needs to be thoroughly analyzed first.
This process is called coding open-ended responses. There are 3 approaches to this.
- Someone internally codes the data manually. If you receive 100-200 responses per month, this is absolutely doable. The big disadvantage here is that there is a high likelihood that whoever codes your text will apply their own biases and simply not notice particular themes, because they subconsciously don’t think it’s important to monitor.
- Outsource to an agency. You can email the results and they would simply send back coded responses.
- The third option is that you can automate the coding. You use an algorithm to simulate the work of a professional human coder.
Whichever way you code text, you want to determine which category a comment falls under. In the below example, any comment about friends and family both fall into the second category. Then, you can easily visualize it as a bar chart.
Code frames can also be combined with sentiment. Below, we’re inserting the positive and the negative layer under customer service theme.
So, next you apply this code frame. Below are snippets from a manual coding job commissioned to an agency.
In the first snippet, there’s a code frame. Under code 1, they code “Applied courses”, and under code “2 Degree in English”. In the second snippet, you can see the actual coded data, where each comment has up to 5 codes from the above code frame. You can imagine that it’s actually quite difficult to analyze data presented in this way in Excel, but it’s much easier to do it using software.
5 ways of visualizing coded data
- Bar charts
- The easiest
- Measure frequency and impact on score
- Theme clouds
- Measure frequency and sentiment
- A better version of word clouds
- Score graphs
- Track numbers or score over time
- Key drivers and impact
- Usually looks like a coordinate area where you can see all the themes grouped according to two attributes which can be quite helpful
- Pivot tables
Do you need software?
Jenine remembers this example: “A client came to me recently, and they were collecting 10,000 comments per year, and they were using Excel. So, you can imagine we became good friends”.
“When we are dealing with data that is impossible to manage manually, because there’s too much of it or because we are afraid to bring in our biases, or if it’s a longitudinal study, there is no other option but to use software”.
So, yes for those reasons, you can see that software is extremely useful.
Above is an example of a very simple way of visualizing the data but it’s also an overlooked one because typically when people visualize open-ended responses they tend to use the word clouds.
Here is a screenshot from Thematic, with two different slices of data. The blue bars are Air New Zealand’s 1 and 2-star reviews and the orange bars are the 4 and 5-star reviews. It’s a great airline, but you can still see the clear difference where pricing stands out as their number one issue, as it’s the most frequent theme among the 1-star reviews. Whereas, when it comes to the 4 and 5-star reviews, it’s the in-flight service and more specifically, their entertainment that stands out.
Now, below is an example of a data set with feedback that we’ve analyzed using a score graph. Here, customers have been surveyed for the duration of one year and the score used to be around 5 but it’s started dropping and it’s now 3 in July.
As you can see here down below, here is a theme cloud where you can see themes that are suddenly more frequent and therefore there are arrows next to them. The colors show the sentiment. So an update to Twitch, one of our customers who hosts an app, actually had a major negative impact on the review score.
Another theme here is using the chat function. People are complaining that they can’t use the chat and you can see the customer comments explaining the issue.
Impact is another bar-chart but a very special kind. I think it’s called a tornado chart because it looks like a tornado. Here, you can see two different customer segments of a financial institution and what drives their score up and down.
Finally, I have a pivot table example where we’re looking at a particular theme for airlines, which is “legroom”. We compare 1 to 3-star Economy vs. First class reviews for four different airlines.
We can see here that Southwest Airlines does not have a first class as it’s a budget airline (therefore the orange bars are excluded) but for the remaining airlines United, American and Delta, we can see some results. United has significantly more people complaining about legroom in Economy class and American has the least amount of complaints. When it comes to First Class, Delta actually has more people talk about legroom than any of the airlines. That’s a nice insight right there.
If you only do one thing – do this
To wrap up, what’s the most important takeaway here? Think about what customers want and what’s in it for them. In my experience, lots of businesses don’t actually think about this when they send out their surveys.
If you can nail the “what’s in it for me”, you automatically solve the incentive decision and the consistency issue for the survey. That, in turn, solves a number of other decisions which means you’ll arrive at a successful outcome of the research.
Other top tips for survey design:
- Report the results back to the customer
- Think about different learning and communication styles
- Make sure you collect data using different methods, accounting for a variety of personality types
- Try to get diverse results
- Do not use Excel to visualize your results
I hope this article has been helpful for your survey, let me know how you go by posting in the comments!
And, if you’d like to see a Thematic demo on your data to see what we can find, simply get a free demo here.
This post was first published here.