How to choose a good sample size for your survey
A fitting sample size is key to extrapolating accurate results from your survey. Too small a sample, and your results will almost certainly be unreliable. Too large, and you’ll have wasted precious resources by oversampling your population.
You don’t need to guess what the correct sample size for your survey should be. Use this straightforward yet proven calculation method to work out the ideal sample and get accurate results from your study.
Your survey sample needs to accurately match the thoughts, opinions, experiences, and backgrounds of the total population you’re trying to study. The appropriate sample size ensures your survey findings are generalizable, meaning they can be usefully applied to the broader population you’re studying.
Say you’re conducting market research on your target audience of millennial dog owners for your brand of gourmet dog food. What is a good sample size for your survey? The answer is one that allows you to draw accurate conclusions about the entire population of millennial dog owners and their opinions and shopping habits around dog food. You want to know the data you’ve collected is a true reflection of the diversity of the population and isn’t negatively affected by having too large or small a sample.
There are risks to choosing the wrong sample size:
You can use either a precision-based or power-based calculation to determine your sample size. For a descriptive/survey-based study, you’ll want to go with a precision-based calculation.
For most types of online surveys, you only need to follow the steps for a precision-based sample size calculation, which we lay out below. If you plan on running statistical tests on your results, you can try this G*Power tool for a power-based calculation and read more about how power-based sample size calculations work.
The total population size is the first figure you’ll need to know before you can work out your sample size. The sample will be based on this number to ensure your survey is capturing a large enough snapshot of the population.
You might already know the exact number of individuals in your population. For instance, if you work for human resources for a Fortune 500 company and you’re conducting an employee engagement survey, you’ll already know how many people work for your company. Or you’re conducting a market research survey, and your total population is adults in South Dakota. You can get that number easily from the latest census data and then take a random sample.
If you don’t know the exact number in your population, you’ll have to make an estimate. Let’s say you want to survey bike owners in U.S. cities. There is no definitive number or place where that data is tracked, but you can research bike ownership and give an estimate to the best of your abilities.
For the sake of our example, let’s say we want to find out whether a new product is ready for the market. The total market for our product is adults in the U.S., so our population size is about 258.3 million people.
How confident do you want to be in your results, and what’s an acceptable margin of error? That’s what you’ll determine with these two figures.
For example, let’s say that you run a study and find that 30% of your respondents answer “yes” to a question. Your margin of error is set at 5, and your confidence level is 95%. You can say that there is a 95% probability that between 25% and 35% of the actual population would have answered “yes” to the same question. So, if you ran the same study 100 times, you would expect that in 95% of those studies, between 25% and 35% of participants would answer “yes.”
Standard deviation is a measure of how much the survey responses vary from the mean. Estimate how much you expect responses to vary from one another to determine the sample size you need.
For example, let’s say you’re looking at whether or not participants like the taste of a new food. In this case, we know that taste is very subjective, so we would expect there to be a higher standard deviation because participants aren’t likely to rate the new food similarly. However, if we were launching a product such as a vacuum cleaner and running a survey to gather customer feedback, we wouldn’t expect responses to vary so much. Objectively, the product either works or doesn’t work, and performance should, hopefully, remain consistent.
If you’re not sure what to estimate, a standard deviation of 0.5 is typical to plug in for an industry survey or other precision-based calculation.
Plug your figures into the sample size formula to calculate the right sample size for your survey.
To use the formula, convert your confidence level to a Z-score. If you’re using a 95% confidence level, you’ll use a 1.96 Z-score.
Look up your Z-score for your confidence level from this table.
The sample size formula you’ll want to use is:
Sample size = (Z-score)2 × standard dev × (1 – standard dev)
(margin of error)2
Using our numbers, our formula would be:
((1.96)2 × 0.5 × (1 – 0.5))
And we would calculate a sample size of 385 participants.
You can also use an online calculator, like this one from SurveyMonkey, to make it even simpler.
Figuring out how many participants you need is only the first step to running a quality survey. You also need to draw those participants from a quality pool of respondents to get results that are truly generalizable to your population.
Today Prolific is turning 5 years old – Happy Birthday to us! 🥳 It's been a remarkable journey so far. 3000+ researchers from science and industry have used Prolific last year, we have 45,000 quarterly active participants, and we've seen 200% year-on-year growth. But we're only getting started. In this post, I'll tell you a little bit about our journey, give credit where it's due!, and tell you about our exciting plans for the future.
Fresh out of YC's Summer 2019 batch, we want to share some of our most interesting learnings. If you're a startup founder or enthusiast and want to learn about product-market fit, growth experimentation and culture setting, you're in the right place!