How to Structure Qualitative and Quantitative User Research Projects


Share on LinkedIn

Product managers spend a significant amount of time talking about getting closer to customers, and the value that comes from understanding user needs. While an industry focus on high-level ideas such as design-thinking and user-centered design are undoubtedly positive, there is less emphasis placed on sharing practical approaches for rapidly obtaining empirical data from a breadth of customers through user research.

Without this more specific discussion of “the how”, new product managers may feel overwhelmed when they look to understand their customers’ needs, or worse yet, they may conduct qualitative and quantitative research only to analyze the resulting data and find a lack of useful and actionable information.

This brief overview of the process of conducting user research will create a framework, outline good practices, and give newer product managers a groundwork from which to work when conducting their own research. The research process should be broken into five basic steps:

  • Framing the Research Objective
  • Conducting Qualitative Research
  • Building the Quantitative Survey
  • Testing the Survey
  • Conducting Research at Scale

Framing the Research Objective

It can be tempting to start your research with a list of questions that you would like customers to answer, but the result of this is often a scattered and meandering set of inquiries that makes it difficult to reach an actionable conclusion. Beginning with a specific and measurable objective will allow you to structure your research to make sure your project ends with clear direction for next steps.

Specific and Measurable Objective:
“Identify customer interest in purchasing company’s new widget.”

From this objective, define a list of areas to explore in more detail in order to determine customer interest of a product. Some of these areas may be:

  • What are customer preferences around pricing and payment options?
  • What are the competitive products?
  • Which attributes are most important to the customer?
  • What does the customer purchase journey look like?

Breaking down the research objective this way gives a clear direction to work towards when developing the specific survey questions, and it also ensures that when data is collected there is sufficient information to determine the customers’ overall interest in the product.

It’s also important to have a clear action, or set of actions, to implement as a result of your research. An example of this could be to begin the research and development of a new product if 20 percent of your statistically significant sample group shows interest, or it could be to change a route to market if you determine that less than 40 percent of your customers would prefer an alternate way of purchasing your products.

The resulting taken action should not be determined by organizational goals, but by establishing this plan prior to conducting your research you will prevent the common pitfall of analyzing your resultant quantitative data and asking “what do we do now based on this information?”

Conducting Qualitative Research

The next step in developing a survey is qualitative research. This is often done in the form of open-ended exploratory interviews with a small number of participants. The objective of qualitative research, as it relates to the survey, is to refine your understanding of the subject matter, and to clarify areas that may be vague or need further development before including them in the quantitative questionnaire.

Qualitative research is relatively time and resource intensive when compared to quantitative surveying, so it’s important to choose the right participants or these initial interviews. While you will usually get useful information from any of the people interviewed, you can maximize the value of the interviews by specifically including several “expert” and “novice” product users in the interviews.

Expert users are helpful because they know exactly what they want. They are often familiar with both your product and other competitive offerings, and they can highlight the critical features and use cases that may make or break the product’s success. Expert users can also point to other solutions or new sources of information that will help you develop your understanding of the needs of your customers.

While expert users are incredibly valuable, they rarely represent an average new user. Novice users; those who have not seen or used products like yours, can also give critical feedback because they are looking at things with “fresh” eyes. Depending on your product, they may or may not look like an average new user, but they can almost always provide helpful insight on the overall usability of your product.

These interviews will help refine questions around each of the areas outlined in your objective. You may also choose to add new sections or to change the approach based on what you have learned. The important thing is to ask probing questions and be open to new information. The more you learn in this phase the better you will be able to structure quantitative questions that tease out real insights from your survey takers.

Building the Quantitative Survey

After interviewing a number of different people and refining your understanding of user interests and concerns, start creating the initial survey draft. This should include refining the categories you probed previously into a list of specific questions. We can use the category of pricing and payment as an example of this.

For the question, “What are customer preferences around pricing and payment options?”, this question refinement may look like the following:

  1. How much would you be willing to pay for the widget?
    1. $0 – $10
    2. $11 – $20
    3. $21 – $30
    4. $31 – $40
    5. More than $40
  2. How do you prefer to pay for widget?
    1. Upfront payment
    2. Multiple payments spread over time
    3. One-time payment when I’m finished using the widget
  3. Would you be interested in financing a widget?
    1. Yes
    2. No

There are many ways to structure questions, but it is important to be as clear as possible, and to limit each question to one specific inquiry. Unclear questions or questions that include multiple facets will make taking the survey much more difficult for participants and will result in less useful data when reviewing the survey responses.

Testing the Survey

Once the survey is developed it is tempting to send it out and start evaluating responses, but some simple testing at this phase will help correct ambiguity, and ultimately make your survey data much more actionable.

A great way to test the survey is to have several target users take it while you observe. Asking participants to talk through their thought process while taking the survey allows you to gather critical insight about misunderstandings or questions where an answer they wanted to select wasn’t provided. Take notes and assume that if participants have problems, others are likely to have the same problems when you conduct your survey at scale.

Conducting Research at Scale

Now that you are happy with the structure and content of your survey, it is time to send it out to a wider audience. There are many services available to connect with specific demographics, but first, there are things to consider, such as:

  • Which user demographics will give me useful responses?
  • How many people do I need to survey to get useful data?
  • What is the cost of accessing the demographics I desire?

Getting actionable information will depend on sending your survey to the right audience. If you’re looking for in-depth information about consumer interest in specific features of a niche product, and you send your survey out to the general public, you may not get useful feedback. However, if you’re looking to determine what portion of the population is interested in your widget, then selecting a random sample may be appropriate.

It is also important to identify the number of respondents needed. If you are looking to get data that is statistically significant and represents a population within a given confidence interval, then you will want to calculate the exact number of respondents needed. Alternately, if you are just looking for general insights from current users about new features being planned, then you may be able to more flexibly choose the number of respondents based on cost or availability.

Cost is often a factor in determining the number of respondents you can poll, but it is also important in selecting which demographics you can reach. Usually, services that reach a general audience will be more affordable, while those that allow you to select specific types of respondents (occupation, income, specific demographic) will be more expensive.

Before publishing your survey, it’s worth double checking the cost of conducting your research to make sure it’s still worthwhile. By weighing the cost of sending out your survey against the value provided by any insights obtained from your data, you can ensure that your survey is still worth conducting. It seems obvious, but if the cost of conducting your survey is greater than the value you get from the information, then it is better to either look for cheaper ways to conduct your research or move on to higher value projects altogether.

Micah Bhakti
Micah Bhakti is a senior product manager at MemSQL, where he focuses on the company’s platform, tools, and managed service offerings. Prior to MemSQL, Micah spent nine years at Intel, most recently building supercomputing software for organizations including NASA, CERN, and The Department of Energy. He has an MBA from the Northwestern Kellogg School of Management. Most importantly, Micah has eaten the best hamburger in North America, twice.


Please use comments to add value to the discussion. Maximum one link to an educational blog post or article. We will NOT PUBLISH brief comments like "good post," comments that mainly promote links, or comments with links to companies, products, or services.

Please enter your comment!
Please enter your name here