Open versus closed questions – which one to use?

by Jennifer Smith

05/06/2024

Table of Contents

What are open questions?
What are closed questions?
The pros and cons of open questions
The pros and cons of closed questions
Additional tips
More resources
References

One of the secrets to creating a survey that collects useful data is the type of questions you ask.

Knowing when to offer respondents a chance to express their own opinions as free text and when to choose questions that allow data to be quantified is a key component of evaluating successfully. These question types are known as open and closed questions, and both have their purposes in data collection.

This article will help you craft surveys to avoid asking questions that are too vague, too restrictive or just plain confusing – making your reporting much easier!

What are open questions?

An example of open questions in the Culture Counts Evaluation Platform

Open questions are the ‘deep divers.’
They provide survey respondents a blank canvas. These questions are designed to let people express their thoughts, feelings and experiences in their own words. This can be incredibly powerful if you are looking to uncover insights that you didn’t even know to ask about.

Examples of open questions:

“What would you like to see at next year’s festival?” (Open Text)

“Please add any comments or feedback about your experience below.” (Open Text)

What are closed questions?

An example of closed questions in the Culture Counts Evaluation Platform

Closed questions are the ‘quick hits.’
They are the survey equivalent of multiple choice tests. These questions give respondents specific options to choose from, making it easy for them to answer quickly and for you to analyse the results.

Examples of closed questions:

“Is this your first time attending this event?” (Yes/No)

“Rate how much you agree or disagree with this statement: It was well thought through and put together” (Slider)

“How did you hear about this event?” (Multi-Select)

The pros and cons of open questions

Open questions can be very engaging, offering the opportunity to gather rich and detailed responses and uncover new issues or ideas. However, the responses can be time-consuming to analyse and not everyone is likely to give a thoughtful or useful answer. It can also be time-consuming for the respondent, increasing your chances of survey fatigue and dropout.

Benefits ✅	Drawbacks ❌
Provide rich detailed responses. Can enable you to hear the voices of your respondents in all their complexity.	Time-consuming analysis. Required to read through and sort the responses, which can be labour-intensive.
Uncover unknowns. Respondent answers can reveal new issues or ideas and are great for exploratory research.	Varied quality of responses. Not everyone will give a thoughtful answer. Some might write a lot, others might give a couple of words.
Engage respondents. They can make respondents feel valued and heard, which can improve the overall survey experience.	Lack of data-driven insights. Free text responses are not able to be turned into charts and graphs, or directly compared year-on-year.

A table depicting the benefits and drawbacks when using open questions

The pros and cons of closed questions

Closed questions are easy to compare and analyse with the added bonus of attracting a high response rate. However, if your answer options aren’t well thought out, your respondents may not have an option that fits their experience, or you may bias their answers.

Benefits ✅	Drawbacks ❌
Easy to analyse. The responses are quantifiable and can be easily turned into charts and graphs.	Limited depth. They don’t let respondents explain the “why” behind their answers.
High response rate. These questions are quick and easy to answer, which can lead to higher completion rates.	Risk of bias. If your answer options aren’t comprehensive or are leading, you might skew your results.
Comparability. They make it easy to compare data across respondents, surveys and over time.	Restrictive. May not include an option for the answer your respondent wants to give.

A table depicting the benefits and drawbacks when using closed questions

Additional tips

The best surveys often use a mix of both open and closed questions.
It is all about balance and knowing what information you need your survey to collect.

Be clear about what data you need to collect and why. This will inform your question selection.

Avoid survey fatigue by keeping the time it takes to answer your surveys to around 3 min. You can do this by asking mostly closed questions with some open questions to provide more depth when necessary.

It is good practice to choose a standard set of questions to form a core of your regular surveys. This allows data to aggregate from survey to survey and be benchmarked. See our blog ‘Why is survey consistency important?’ for more info.

Using templates is a great starting place for designing your survey. The Culture Counts Evaluation Platform has a range of best-practice templates. Take a look to gain an overview of questions to include.

Give respondents the option to answer ‘none of the above’ or ‘other (please specify)’ in closed-format questions. If a high number of respondents answer ‘other’, you may be able to extract a theme from these answers and add new options to the next survey.

A word cloud may be one way to visually display information from short answer open questions.