Statistical significance is the gold standard of reliability in research. But what does it really mean in customer insights research? And can you make informed decisions without it?
What is statistical significance?
The first question to come up in any customer research readout is always, “Are these results statistically significant?” Over the years, decision makers have started throwing the phrase “statistically significant results” around as though it is synonymous with “reliable insights.” In this article I explore leaders’ obsession with statistical significance and why this oversimplification can a dangerous for your decision making.
Statistical significance is a method used to determine whether an observed effect or difference is likely due to chance or is a true effect. The different levels of statistical significance (typically 90%, 95% or 99%) tell us that a result can be trusted and is not due to random chance. In other words, at a 95% confidence interval: if you repeated the study 100 times, you would reach the same conclusion 95 out of the 100 times. Clearly, it is easy to see how powerful statistical significance can be in understanding the accuracy, precision, and the extent to which we can extrapolate data findings. But what if achieving a sample size large enough to establish statistical significance isn’t feasible in your situation?
In practice, customer insights research isn’t done in a lab or with unlimited resources. We therefore must adjust our approach: balancing budget constraints, business timelines, and the reliability of the insights.
Imagine a scenario in which a client needs to understand an audience who is very difficult to reach. Perhaps it’s B2B IT decision makers at $100M+ companies or maybe it’s pediatric doctors in the San Francisco area. Either way, this audience is both very difficult to find and very expensive. It will take a very large budget and an extended timeline to collect data from a large sample of these individuals. Working with research panels, the biggest sample that is feasible may be n=200. With this amount of data, it will be very difficult to establish statistical significance at any confidence level. However, does that mean that we can’t learn anything from these 200 opinions? No!
Statistical significance is a powerful tool, but it should not be viewed as black and white – or reliable and not reliable. The goal of customer insights research is to understand the consumer behaviors, preferences, attitudes, and needs that consistently emerge within our audience. As a result, the need for statistical significance in consumer insights depends on the specific goals of the research and the nature of the data being analyzed. There is still a lot to be learned from smaller samples if we take a strategic, thoughtful approach. Below, I have outlined a quick guide to achieve actionable, reliable insights, even when a large sample is not available.
1. Develop a thoughtful research methodology and analysis plan.
This starts with a comprehensive research brief to understand the intent of the research and collect important background context.
Next, it is critical to sit down with your stakeholders to establish key research objectives and set expectations for the results. Some critical questions to ask in these sessions could be:
- What triggered the need for this research?
- What will these insights ultimately be used for?
- At the end of the day, what would a successful research study look like?
Your research will only be as good as your research objectives, so make sure these are solidified and agreed upon by all stakeholders.
With the research objectives in hand, we can then design the study to meet these objectives. Knowing what we need to achieve allows us to thoughtfully craft research questions that tie directly to the key objectives.
When sample size is constrained, it can be helpful to structure questions from multiple angles. This gives analysts on the back end an opportunity to track the consistency of responses. For example, asking the importance of specific product attributes independently using a Likert scale, or by employing unstructured methods such as open-ended questions and follow-up probes. These methods can provide more detailed and nuanced information that may not be typically captured by a quantitative survey which can be very powerful in identifying patterns or trends in the audience.
2. Plan the right audience definitions and sampling plan for your objectives.
Your sampling plan will be dependent on a handful of factors, for example:
- What are your budget constraints?
- How long of a timeline do you have to deliver insights?
- How niche is the ideal target audience?
- How critical are the decisions being made based on the research output?
First, you need to define your audience. This will determine whose voices we hear from in the research. It is important to sit down with stakeholders and establish exactly what qualifications a relevant audience must have. Furthermore, within that audience, are there any specific sub-audiences we need to account for? For example, if our audience is orange juice purchasers in the last 6 months, we may want to set quotas around age groups or online vs in-store purchasers.
Next, work with a trusted panel provider who can design a panel strategy to minimize bias. Focus on quality over quantity: you should recruit high-quality respondents who are engaged and satisfy your audience definitions. Additionally, it is important for your sample group to be representative of the broader population, so balancing to match the census is a good way to remove potential bias from age, gender, ethnicity and region. A well-designed sampling plan with appropriate controls and quality assurance measures can provide reliable results even if statistical significance is not reached.
3. Use critical thinking to make informed judgements
Analyzing survey data requires careful consideration and interpretation of the data. By using appropriate statistical techniques, identifying persistent trends, and considering qualitative data, you can draw meaningful insights from even a small sample size.
Start by choosing appropriate statistical tests: With a smaller sample size, it is important to choose statistical tests that are appropriate for the data and the research question. You can also report on effect sizes – or measures of the magnitude of the difference between groups or the strength of the relationship between variables. Reporting effect sizes can help to provide a more meaningful interpretation of the results, particularly for small sample sizes where statistical significance may not be sufficient to draw conclusions.
Look for consistent patterns in the data. Even with a small sample size, it may be possible to identify patterns in the data. You can use descriptive statistics such as frequencies and percentages to identify real trends and patterns in the data as well as looking at the story across the entire data set rather than at each question independently.
Finally, include insights from both qualitative and unstructured data. By including both quantitative and qualitative information, an analyst can see the bigger picture and provide additional context to help explain the results of your survey.
As always, transparency about the power as well as constraints of the study is important throughout the research. As a researcher, you need to interpret the results cautiously and to acknowledge the limitations of the small sample size up front with your audience. Should you feel it is needed, you can always recommend an additional study to dive into any ambiguous results.
So, how significant is statistical significance in customer insights research?
While no one can dismiss the power of statistical significance in your customer insights research toolkit, it is just that – one tool at our disposal as researchers. It is not a deal breaker for a successful study. By thoughtfully setting up your research approach, planning a robust sampling plan and critically exploration the data for consistent patterns, you can achieve actionable, reliable insights, even when statistical significance is not feasible.
