Clients ask me these questions all the time:
- “For a precise analysis, won’t it be expensive to examine every customer comment?”
- “How can we get most of our customers to take our survey?”
But here’s the truth.
Examining every comment or getting feedback from every customer is simply unnecessary for achieving accurate data. In fact, if you fail to sample your data, you may inadvertently skew your results.
- Perhaps you have thousands of customer comments. We only need to tag about 370 of them (per customer population) to make an objective analysis.
- Likewise, you may send your survey to thousands of customers, but we only need a fraction of them to respond.
How is that possible? The answer is the statistical method known as sampling.
If you already know everything about sampling, use our handy survey sample size calculator below. It shows you how much evidence you need from any given population to get reliable and replicable results.
However, keep reading if you need a primer on the basics of statistical sampling.
Survey Sample Size
When done correctly, statistical sampling results in accurate findings for your overall population.
By examining a small subset of data and calculating the survey sample size, we can forecast how a population feels and thinks, no matter how large that population may be, within a small margin of error. Does that sound like too much science? Don’t worry; it’s really pretty simple.
It Boils Down to …Spaghetti Sauce?
Imagine you’re making a big pot of spaghetti sauce. Maybe it’s four quarts; maybe it’s eight quarts; maybe it’s 12 quarts. At a certain point, it doesn’t matter how much sauce you’re making; you only need a small bite’s worth to test whether you added enough salt. Of course, this assumes that you mixed your sauce well enough, and we’ll explain that part soon.
Not that confusing, right? And our handy survey sample size calculator lets you know exactly how big your “bite” should be based on the size of your dataset. If you might want to calculate your survey sample size in the future, bookmark this page.
Usually, you should set the margin of error to 5%. That’s a recognized norm, but if you need to be more exacting, use a 2% margin of error. If you merely need directional findings, an 8-10% margin of error could be fine.
Next, we’ll unpack margin of error and the other terms you need to know to use our calculator.
Sampling Simplified
OK, what is the sampling process? Let’s assume you have an Excel spreadsheet with 3,000 customer comments. You’d like Text Analysis to understand what your customers said, so you come to us. (We can already tell you’re smart!)
First, we’ll examine your data to see if your customer base is the same demographic or whether it includes diverse groups. For instance, Original Equipment Manufacturers may have very different feedback than Distributors and End-Users.
We only need to look at a sample of a few hundred comments from each sub-group to characterize each population’s perceptions accurately.
For sampling to work correctly, two things are necessary:
- your sample must be representative of the entire population, and
- it has to be large enough.
Survey Sample Size Biases
As Nate Silver, statistician and founder of FiveThirtyEight.com says, “Wherever there is human judgment, there is the potential for bias.”
Here’s a super simple example of a bias at work:
- Imagine your population is candy: 500 red M&M’s and 500 blue M&M’s.
- But your sample consists of eight red M&M’s and 270 blue M&M’s.
- Your sample is biased, and your results will be skewed.
There’s more than one way to mess up sampling. Here are a few common forms of sample bias:
- Selection bias: This error occurs when researchers inadvertently exclude portions of the population they intend to study. For instance, if you conduct an online survey, you’ll exclude those without internet access.
- Bias through convenience: Voluntary surveys can be convenient, but they only include data from the people who were willing to take the survey in the first place. They exclude all those who are part of the target population but didn’t sign up for the survey. Perhaps they didn’t have the time, the interest, or anything to say that they think is worthy of your attention.
- Nonresponses: This happens when many of your participants don’t respond to your survey. If you fail to get enough responses, you won’t be able to generalize the results to the overall target population.
Sample Size Quality Control
How do you avoid bias and make sure your sample represents the overall population? Here’s a general outline of the process:
Step 1: Define your population.
Describe how you will bound your population; think through your inclusions and exclusions carefully.
Step 2: Choose your sampling method.
Here are a few common sampling methods:
- Simple random sampling using a random number generator or drawing lots.
- Stratified sampling: divide the population into subgroups based on certain characteristics and then randomly sample from each subgroup based on its size. This is a good method to ensure that all subgroups are represented.
- Systematic sampling: choose every nth individual from a list of the population, with a randomly chosen starting point.
Step 3: Calculate your survey sample size.
To calculate the correct sample size for your given population, you need to know:
- Margin of error: The lower the margin of error, the more accurate the results. A margin of error of 5% is often used in polling studies. Remember that spread is really 10% because this means plus OR minus 5%. Basically, the higher the stakes, the smaller your margin of error should be. If you are building a rocket for NASA, then your margin of error needs to be low.
- Confidence level. Statisticians often refer to a confidence level, commonly choosing 95%. What does that mean? Let’s assume a confidence level of 95%. If you were to repeat your analysis or survey over and over again, the results would match the results from the actual population 95% of the time, or 19 times out of 20.
When you combine your margin of error (for example, 5%) and your confidence level (for example, 95%), you can say that you are 95% sure that the true measurement of the general population is within 5% of your survey results.
Small Sample Size, Big Results
So, what does all this have to do with survey sample sizes? Once you know your margin of error and confidence level, you can calculate the sample size you need to achieve statistically valid results.
Here’s the surprising part: the survey sample size is probably a lot smaller than you might assume. Assuming a margin of error of 5% and a confidence level of 95%:
- If your population is 100, you’ll need a sample of 80.
- But if your population is 1,000, you’ll only need a sample of 278.
- Even when the population is 1,000,000, you only need 385 measurements to get accurate results.
So, when clients come to us with thousands of customer comments, we need to tag fewer than 400 of them per population to provide factual insights. Time saved is money saved!
However, there are situations where we can make a strong case for NOT sampling and it’s better to hear from every customer or tag every comment.
To discuss the right sample size for your data—and if sampling makes sense for your project, get in touch.
