Are the results from the sample I surveyed an accurate reflection of my whole audience? How likely is it that I would get the same results if I surveyed everybody who attended?
These are questions you may ask yourself when looking at the data from a sample of a population, whether it be audience members or event attendees. Generating a statistically significant sample means you can be confident your survey results reflect the opinion of your wider audience.
A recent case study from KAGE discusses the common challenge that organisations can face in capturing large survey samples. One of 23 organisations that participated in the Creative Victoria pilot, KAGE found that they “didn’t always get enough people to undertake the survey, which would make the results more statistically representative of our audiences.” This can be more challenging for smaller events or organisations, and is often coupled with uncertainty in determining what is a statistically significant sample.
In this blog we will run through how to determine and report on significance, and tips on how to ensure you have the best chance of achieving a representative sample of your audience.
Margin of Error and Confidence Level
How well the sample represents the population is gauged by two important statistics – the margin of error and confidence level.
The margin of error indicates the likelihood that the result from a sample is close to the result you would get if the whole population had been surveyed. It determines the range of values above and below the sample statistic that are likely to occur. Margins of error under 5% can be considered reasonable representations of the opinion of the population.
The confidence level represents how often the population would pick an answer that lies within the margin of error. It ultimately tells you how sure you can be of accuracy. Applying a 95% confidence level is common practice for business reporting and allows you to be confident that 95/100 times the result would fall within the margin of error range when surveying the entire population.
The margin of error for all dimensions, at a 95% confidence level, is now available to see in real time in the Culture Counts Reporting Dashboard. Contact us to input your estimated audience size for an evaluation to calculate and display this chart.
Tips and Insights
Sample Size
How many responses do I need to get?
There is no minimum number of responses required in order to obtain a significant sample, however it is best to try to capture as many responses as possible. Achieving larger samples enables you to be confident that the average scores and opinions of the survey group are representative of the total audience.
We generally suggest aiming for 100 responses but this will depend on your total audience size and variation in opinion. The bigger your total audience, the more responses you should aim for. Variance in response is almost impossible to predict, but if you feel your event may polarise audiences you should aim to get more responses.
A bigger sample is good, but the sample size you need may not always be as big as you think. If the variance in feedback is minimal, even a small sample from a large total population can yield statistically significant results.
Sample Selection
Who should I be asking?
It is critical that survey respondents be chosen randomly so the results are not biased toward particular members of the population. Using a mixture of distribution methods can be an easy way to maximise your responses as well as ensuring a wider mix of audience members. Surveys can be distributed in many ways, including social media, email, and employing interviewers with tablet computers. If conducting face-to-face ‘interview’ surveys we encourage interviewers to approach people of all demographics to ensure a representative sample.
Remember that evaluations with a smaller sample can still produce very useful audience data, and engaging with audiences by asking for feedback is beneficial regardless of whether the sample is deemed statistically significant. So never hesitate to capture feedback and contact us if you would like to evaluate your next event using Culture Counts.
Image Source: Melbourne street White Night – Nils Versemann
