It can be easy to make the assumption that the bigger the sample size, the better your results will be. This assumption can make data collection feel like an overwhelming task!
But, the great thing about data is that we can extrapolate from small samples and still get rigorous results. We have a few tips to help you understand sample sizes and how to apply this knowledge when planning data collection for your next evaluation.
A core feature of statistics is the ability to infer information about a large population from a smaller-sized sample. We see this every day; from political polling, to clinical trials of medicines. The use of a sub-set is a convenient method of representation when working with large populations.
However, this begs the question – how do we know if and when a sample can be a reliable representation of a larger population?
There are some easy steps you can take prior to data collection to take the stress out of knowing what will constitute an adequate sample size for your event. We also have some easy ways you can encourage responses to your survey.
In this blog we will explain the following:
- Statistical concepts such as population, confidence levels and margin of error.
- How we have built these concepts into the Culture Counts Evaluation Platform.
- Practical next steps for you to implement your learnings.
What do we mean by population?
In evaluation terms, ‘population’ often refers to the entire group of event attendees or participants, not the entire population of a geographical area.
For example, the ‘population’ of an event could be 100,000 attendees but the sample is the 400 people who responded to a survey. The goal of the sample is to collect enough responses to give an accurate representation of the sentiment and demographics of the population – this is where margin of error is important!
Let’s consider two examples to gain a better understanding of why confidence levels and the margin of error are included.
Example 1: Organisation A
Total attendees (population): 100,000
Survey response count (sample): 400
Organisation A ran an event with 100,000 attendees, and received 400 responses to their event survey. The 400 responses to the survey are used to generate mean scores across the measured dimensions, but how sure can we be that those mean dimension scores are truly representative of all attendees? Well, at a 95% confidence level, we get a 4% margin of error for this population/sample ratio.
This means that we can be 95% confident that if all 100,000 attendees completed the survey, then the mean dimension score of the 100,000 attendees would lie within 4% of the mean dimension score from the sample surveyed.
In statistics, a margin of error less than 5% is considered an accurate representation of the total population. This means Organisation A can confidently conclude that these results are an accurate representation of the audience’s perception of the event.
To ensure that the Margin of Error is calculated accurately in the Analytics Dashboard, users can input the attendance number for an event once it has concluded via ‘Evaluation Properties’. This will result in a rigorous confidence interval and margin of error for dimension results in the Analytics Dashboard.
Example 2: Organisation B
Total attendees: 100,000
Survey response count: 80
Organisation B ran a similar event attended by 100,000 attendees but received fewer responses than Organisation A, with only 80 people completing the survey. At a confidence level of 95% with a lower response count, the margin of error is calculated to be 11% .
With a margin of error more than double that of Organisation A’s event survey, and well above the 5% threshold, we cannot be confident that the results received are an accurate representation of the audience’s perception of the event.
Next steps
- Have an expected attendee number ready – this can be a range, but this number is important for calculating the ideal sample size!
- Use a sample size calculator to input your attendee (population) figure, and add in a confidence level of 95% – this is the percentage we recommend.
- Plan your communications and how you will be gathering feedback – diversify your distribution methods to ensure you are getting responses from varied respondent types.
- Keep track of your responses via the Analytics Dashboard – if your sample size has fallen short of the goal you will know to send out a final push via social media or email! The sooner you get the survey out the better, as most responses are collected within the first few days of an event concluding. Doing a final push for responses within a week or so can help get you over the line.
- Don’t be discouraged if you haven’t reached your sample size goal! Data from your survey might actually help inform your next evaluation – for example, look at the responses for questions about marketing and see if you can adapt for next time.
Tips
Finding it hard to collect responses? Here are some ways you can increase your sample size:
- Create a poster with a QR code so people can take the survey on their smartphones
- Email a survey link to ticket holders or your member database
- Add a survey link to your public-use devices (great for libraries!)
- Incentivise respondents to complete your survey
Article written by Culture Counts’ Data Analysts Jamie McCullough and Riley Chappell; and Client Manager Shelley Timms.
